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EXECUTIVE  SUMMARY 


Department  of  Defense  (DoD)  energy  success  is  measured  against  mandated  goals  for  energy 
reduction  and  sustainable  facility  management.  In  order  to  make  consistent  and  well-informed 
decisions  across  its  entire  portfolio  of  buildings,  DoD  has  a  critical  need  for  a  consistent,  scalable 
approach  to  evaluating  energy  consumption  of  existing  facilities,  to  compare  tradeoffs  between 
energy  conservation  measures,  and  to  identify  facilities  that  are  in  greatest  need  of  improvement. 

In  the  last  several  years,  it  has  become  increasingly  evident  that  existing  methods  of  simulating 
and  estimating  energy  use  in  buildings  require  highly  trained  engineers  to  spend  significant  time 
constructing  energy  analysis  simulations.  Shortcomings  of  past  approaches  included  labor- 
intensive  data  inputs,  the  need  for  subject  matter  experts  to  operate  the  modeling  systems,  and 
the  inability  to  model  the  DoD  building  inventory  in  a  timely  or  cost  effective  way.  Autodesk 
began  looking  at  ways  to  combine  various  data  collection  methods,  best  practices  and  software 
tools  to  address  this  problem,  and  the  idea  of  Rapid  Energy  Modeling  (REM)  was  conceived. 

Overall,  the  goal  of  the  demonstration  was  to  evaluate  REM  workflows  and  performance  by 
comparing  simulated  to  actual  building  energy  consumption  and  investigate  the  scalability  of 
REM  workflows  for  the  DoD.  This  project  demonstrated  that  the  REM  workflow  quickly 
captures  and  utilizes  information  on  operations,  geometry,  orientation,  weather,  and  materials, 
generating  Three-Dimensional  (3D)  Building  Information  Models  (BIM)  guided  by  satellite 
views  of  building  footprints  and  simulating  energy  use  patterns.  In  addition,  the  project 
demonstrated  the  application  of  simulated  Energy  Conservation  Measures  (ECM)  on  a  subset 
population  of  buildings  to  understand  effective  ways  to  reduce  their  energy  consumption.  The 
REM  technology,  including  the  ECM  capabilities,  uses  whole-building  energy  simulation 
algorithms  driven  by  the  Department  of  Energy  (DOE)  2.2  engine  for  energy  analysis. 

REM  was  applied  to  a  sample  of  23  DoD  buildings  across  eight  locations  and  representing  seven 
building  types.  The  simulated  and  actual  building  energy  data  was  analyzed  by  energy  type 
(electricity  and  natural  gas)  and  energy  use  intensity  (EUI)  and  further  segregated  by  building 
type.  The  results  show  that  the  models  for  offices  and  specialty  use  buildings  performed  better 
than  models  for  barracks,  where  variable  occupancy  did  not  match  model  assumptions. 

Quantitatively,  a  primary  performance  objective  was  to  have  REM  electric  and  natural  gas 
estimates  come  within  <  10%  of  actual  utility  information  (90%  average  accuracy).  Aggregate 
results  indicate  average  accuracy  of  81.88%  for  predicting  electric  consumption  with  a  mean 
absolute  percentage  error  of  18.12%  (Table  7;  Appendix  B),  considered  to  be  a  good  forecast 
according  to  published  criteria  (Lewis,  1982).  Natural  gas  and  combined  EUI  predictions  were 
on  average  58.20%  accurate  and  77.56%  accurate  respectively,  considered  reasonably  accurate 
(Table  6).  The  demonstration  produced  margins  that  while  outside  the  target  range,  were  still 
within  the  range  of  useful  forecasting  values  (Table  6),  with  strong  correlations  in  energy  use 
curves  for  many  buildings. 

Qualitatively,  the  training  completed  indicates  that  the  project  meets  the  performance  objectives 
showing  that  DoD  participants  can  learn  the  workflow  and  begin  creating  and  analyzing  using 
REM  in  less  than  one  day.  Participants  also  indicate  a  high  level  of  satisfaction  with  the  REM 
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workflow.  Preliminary  results  indicate  that  energy  models  can  be  completed  in  less  than  3  hours 
after  the  process  is  learned  (the  performance  objective  was  2  days). 


A  significant  number  of  considerations  were  uncovered  that  help  guide  the  refinement  of  the 
REM  process  in  the  future,  including  data  gathering  and  model  sensitivity.  Additionally,  the 
quality  of  the  DoD  building  meter  data  was  not  as  high  as  expected  before  the  start  of  this  project 
and  as  a  result,  there  may  be  discrepancies  in  comparison  of  simulations  to  the  meter  data. 

While  the  REM  process  and  reports  do  not  mirror  traditional  audits,  the  workflow  has  potential 
benefits  in  that  it  can  be  implemented  by  DoD  personnel  directly.  It  is  difficult  to  do  a  direct 
comparison  to  cost  or  time  savings  with  traditional  audits  as  there  is  not  complete  overlap  in 
capabilities,  but  results  indicate  that  REM  can  yield  >90%  savings  in  time  and  cost  compared  to 
traditional  American  Society  of  Heating,  Refrigerating  and  Air  Conditioning  Engineers 
(ASHRAE)  Level  2  auditing  approaches,  with  the  added  benefits  of  computer  simulation 
characteristic  of  Level  3  audits.  REM  analysis  can  be  completed  in  less  than  one  hour,  with  up  to 
two  additional  hours  that  may  be  required  for  data  collection.  The  modeling  process  requires 
minimal  training  or  expertise  and  has  been  taught  to  DoD  staff  in  less  than  one  day  during  this 
demonstration  project. 

The  results  of  this  study  indicate  that  REM  can  meet  the  need  of  the  Energy  Independence 
Security  Act  (EISA)  2007  data  reporting  requirements  as  well  as  support  government  policy 
including  Executive  Order  13423.  REM  provides  the  DoD  with  a  way  to  quickly  establish 
building  geometry,  scale  energy  analysis  of  the  existing  building  portfolio,  visualize  end-use 
breakdowns  of  energy  consumption,  compare  tradeoffs  and  potential  energy  savings  between 
energy  conservation  measures  automatically,  identify  facilities  that  are  in  greatest  need  of 
improvement,  and  enhance  scalability  of  energy  evaluations  and  retrofits. 


Quantitative  Benefits 

Average  Accuracy  Comparison  to  Historic  Utility 
Information 

EUI  electric  average 

81.88% 

EUI  natural  gas  average 

58.20% 

Combined  EUI  average 

77.56% 

Application  of  design  alternatives  to 
model  potential  energy  savings 

Energy  savings  greater  than  30%  achieved  on  three 
out  of  five  buildings. 

Time  and  cost  to  create  energy  models 

Cost  savings  of  over  95%  and  time  savings  of  90- 
95%  compared  to  ASHRAE  Level  2  audits. 

Qualitative  Benefits 

End  User  Effort 

Ease  of  learning  REM  process 

Less  than  one  day 

Effort  to  create  a  REM 

3  hours  per  building  with  added  benefit  of  auto¬ 
generation  of  multiple  simulations  to  explore  and 
prioritize  ECMs 
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1.0  INTRODUCTION 


1.1  BACKGROUND 

Current  building  energy  assessment  methods  for  existing  buildings  are  expensive,  laborious,  time 
consuming  and  require  a  high  level  of  technical  sophistication,  experience  and  expertise  that 
takes  years  to  establish.  In  short,  typical  building  energy  assessment  methods  are  not  scalable 
across  a  large  number  of  buildings. 

The  energy  consumed  by  facilities  owned  and  operated  by  the  Department  of  Defense  (DoD) 
accounts  for  approximately  80%  of  the  total  energy  used  by  Federal  buildings  (DoD,  2005). 
However,  determining  information  about  the  energy  use  on  military  bases  is  challenging,  as 
buildings  historically  have  not  been  metered  individually.  Due  to  data  quality  issues  and  lack  of 
access  to  information,  facility  managers  or  resource  efficiency  managers  have  difficulty 
managing  their  building  energy  footprints  and  prioritizing  their  energy  retrofit  budgets 
effectively. 

DoD  energy  success  is  measured  against  mandated  goals  for  energy  reduction  and  sustainable 
facility  management.  In  order  to  make  consistent  and  well-informed  decisions  across  its  entire 
portfolio  of  buildings,  DoD  has  a  critical  need  for  a  consistent,  scalable  approach  for  evaluating 
energy  consumption  of  existing  facilities,  to  compare  tradeoffs  between  energy  conservation 
measures,  and  to  identify  facilities  that  are  in  greatest  need  of  improvement. 

Evaluation  of  baseline  energy  use  and  identification  of  opportunities  for  improved  building 
performance  are  top  priorities  for  decreasing  carbon  emissions,  reducing  energy  costs  and 
enhancing  energy  efficiency.  Additionally,  energy  security  and  regulatory  mandates  are  key 
drivers  of  energy  efficiency  retrofits  across  the  DoD.  Typical  approaches  for  rapidly  assessing 
and  benchmarking  energy  usage  and  evaluating  proposed  energy  retrofit  measures  are  not  precise 
and  often  fail  to  acknowledge  the  complexity  of  buildings  and  building  performance.  Interrelated 
factors,  such  as  building  orientation,  location,  operational  use,  and  structural  idiosyncrasies  can 
all  influence  energy  use  and  the  effectiveness  of  retrofit  decisions  on  reducing  energy  usage  and 
energy  costs.  More  comprehensive  energy  auditing  techniques,  such  as  American  Society  of 
Heating,  Refrigerating  and  Air  Conditioning  Engineers  (ASHRAE)  level  audits  are  costly,  time¬ 
intensive  and  require  a  high  level  of  expertise. 

To  address  these  challenges,  Autodesk  executed  a  demonstration  of  Rapid  Energy  Modeling 
(REM)  workflows  that  employed  building  information  modeling  (BIM)  approaches  and 
conceptual  energy  analysis.  The  project  investigated  the  hypothesis  that  REM  is  a  viable  and 
scalable  method  for  generating  accurate,  rapid  and  cost-effective  estimates  of  energy 
consumption  for  DoD  buildings.  The  demonstration  was  a  pilot-scale  operation  over  a  1-year 
period  using  a  population  of  35  buildings  and  an  analyzed  sample  of  23  buildings. 

The  benefit  of  this  technology  is  that  it  puts  a  viable  building  energy  assessment  method  in  the 
hands  of  DoD  installations.  On-site  personnel  can  reasonably  learn  and  use  this  approach  to 
prioritize  the  energy  management  decisions  needed  at  their  installation.  This  technology  can 
dramatically  decrease  the  time  it  takes  to  understand  the  energy  performance  of  DoD  buildings. 
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1.2  OBJECTIVE  OF  THE  DEMONSTRATION 


The  project’s  objective  was  to  investigate  REM  to  determine  if  the  workflow  is  capable  of 
producing  useful,  rapid  and  cost-effective  estimates  of  energy  consumption  for  DoD  buildings. 
REM  would  then  provide  DoD  numerous  benefits,  including  the  ability  to:  meet  federal 
mandates,  increase  energy  security,  enhance  the  ability  to  prioritize  energy  efficiency  retrofit 
projects,  track  energy  use  reductions,  and  manage  facilities  in  new  and  cost-effective  ways. 

The  overarching  objective  of  the  field  demonstration  was  to  provide  lightweight  BIMs  and  an 
easily  scalable  REM  methodology  for  estimating  energy  intensity  in  DoD  buildings,  identifying 
buildings  that  would  be  most  responsive  to  improvements  and  exploring  various  Energy 
Conservation  Measures  (ECM)  for  buildings.  The  technology  demonstrated  included  a  workflow 
for  creating  digital,  three-dimensional  (3D)  models  of  buildings  from  publicly  available  satellite 
or  aerial  imagery.  The  process  captures  existing  building  geometry,  appends  operational 
characteristics  as  well  as  local  weather  data  to  generate  3D  models  to  estimate  the  energy  use  of 
the  modeled  buildings.  The  research  objectives  in  this  demonstration  include  a  comparison  of  the 
REM  generated  energy  use  simulations  to  historical  metered  data.  This  validation  was  carried  out 
to  provide  confidence  in  the  REM  methodology.  Also  validated  are  the  time  and  cost  to  produce 
results  with  this  REM  approach  as  well  a  comparison  of  cost  requirements  for  other  approaches 
such  as  energy  auditing.  In  addition,  this  demonstration  validated  the  acceptance  and  use  of  the 
REM  technology  by  DoD  personnel  at  installations. 

1.3  REGULATORY  DRIVERS 

The  following  existing  or  anticipated  federal,  state,  or  local  regulations  or  DoD  directives  have 
resulted  in  a  need  for  a  new  technology  such  as  REM: 

•  Energy  Policy  Act  (2005)  -  Requires  that  federal  buildings  be  metered  “ to  the  maximum 
extent  practicable.  ’’  Despite  the  mandate,  the  majority  of  DoD  buildings  are  still  without 
meters.  REM  processes  can  help  the  DoD  evaluate  and  benchmark  energy  use  in 
buildings,  assist  in  determining  which  buildings  are  practical  to  meter,  identify  buildings 
with  meters  that  are  not  functioning  well,  and  identify  poorly  performing  buildings. 

•  Energy  Independence  and  Security  Act  of  2007  (EISA,  2007)  -  Sets  a  target  for  the 
government  to  reduce  its  energy  and  other  resource  consumption  by  30%  by  2015 
compared  to  a  2003  baseline.  Additionally,  EISA  calls  for  energy  and  water  audits  for 
25%  of  facilities  annually  and  all  appropriate  facilities  on  a  4-year  cycle  (AEMR,  2010). 
Using  REM  processes,  the  project  team  conducted  rapid  audits  of  DoD  buildings  and 
investigated  ECMs  to  achieve  reductions  in  energy  use  on  a  subset  of  five  buildings. 

•  Executive  Order  13423  Strengthening  Federal  Environmental,  Energy,  and 
Transportation  Management  (2007)  -  Encourages  continuous  improvement  in  the  areas 
of  energy  efficiency,  renewable  energy,  water  conservation,  and  sustainable  building. 
Models  produced  through  the  REM  process  can  be  updated  and  accessed  continually, 
thus  allowing  energy  managers  to  continuously  explore  energy  saving  and  energy 
generation  opportunities. 
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•  Guiding  Principles  for  Federal  Leadership  in  High  Performance  and  Sustainable 
Buildings  -  Calls  for  30%  reduction  in  energy  costs  for  new  construction  and  20% 
reduction  in  major  renovations.  REM  processes  can  be  used  to  investigate  renovations  to 
meet  these  energy  cost  reduction  targets  and  provide  a  higher  level  of  customization  than 
benchmarking  without  the  time  and  cost  associated  with  ASHRAE  or  investment-grade 
energy  audits. 
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2.0  TECHNOLOGY  DESCRIPTION 


2.1  TECHNOLOGY  OVERVIEW 

The  demonstration  defines  a  process  to  capture  existing  building  geometry  using  satellite  photos. 
The  operational  characteristics  of  the  building  are  appended  to  the  geometric  model  and  local 
weather  data  to  generate  energy  models  that  can  quickly  predict  the  energy  use  of  the  modeled 
buildings.  This  information  can  help  asset  managers  determine  which  buildings  are  performing 
poorly  compared  to  predicted  energy  use. 

The  REM  process  involves  the  following  technologies  (Figure  1): 


•  Autodesk®  Formlt  software  is  an  iOS  and  Android  operating  system  application  to  create 
3D  models.  Formlt  captures  existing  building  conditions  using  satellite  images  from 
Google  and  allows  users  to  create  a  3D  geo-referenced  building  model  while  in  the  field. 

•  Autodesk®  Revit  is  a  BIM  software  application  with  integrated  energy  and  carbon 
analyses  driven  by  Green  Building  Studio  (GBS)  and  Department  of  Energy  (DOE)  2.2. 

•  Autodesk®  Vasari  software  is  for  creating  building  conceptual  models,  with  integrated 
energy  and  carbon  analyses  driven  by  GBS  and  DOE  2.2. 

•  Autodesk®  GBS  is  a  web  service  that  performs  whole  building  energy  analysis  using  the 
DOE  2.2  engine. 


The  REM  workflow  involves  three  stages  involving:  (1)  capture  of  existing  conditions, 
(2)  conceptual  modeling  of  building  masses  using  Formlt,  Revit  and  Vasari,  and  (3)  comparative 
analysis.  The  energy  results  of  these  building  analyses  are  represented  as  annual  energy  use  for 
natural  gas  and  electric,  monthly  and  annual  cost,  monthly  energy  use  and  energy  use  intensity 
(EUI)  (Figure  1). 


Figure  1.  REM  Technology  Components. 
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2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 


Current  methods  and  high  costs  for  energy  audits  may  limit  their  practicality  for  implementation 
across  the  DoD,  and  less  expensive  benchmarking  approaches,  such  as  Energy  Star  and 
Commercial  Building  Energy  Consumption  Survey  (CBECS),  do  not  provide  building-specific 
detail  or  identify  opportunities  for  savings. 

Alternative  technologies  include  several  energy  modeling  graphical  user  interface  front  ends  to 
generate  building  geometry  and  apply  energy  modeling  attributes.  It  was  beyond  the  scope  of 
this  project  to  understand  the  relative  technical  merits  of  these  applications.  Several  of  these 
alternatives  are  available  from  the  DOE  (NREL,  2013). 

The  REM  workflow  for  energy  assessments  can  provide  advantages  by  offering  a  level  of  detail 
not  obtained  through  benchmarking  and  with  significantly  less  cost  than  energy  audits.  A 
limitation  is  that  REM  does  not  provide  the  detail  of  investment  grade  energy  audits  and  does  not 
cover  some  aspects  of  a  Level  2  energy  audit  (such  as  equipment  inventories  and  estimating 
costs  for  ECMs),  although  it  does  include  computer  simulation  often  part  of  Level  3  audits.  The 
detailed  attributes  typically  required  for  the  Level  1  or  2  energy  audits  are  not  based  on  an 
understanding  of  the  relative  sensitivity  of  these  attributes  to  energy  model  performance,  so  it  is 
difficult  to  say  how  much  of  a  limitation  it  is  to  simply  allow  some  attributes  to  be  defined  with 
default  values.  Where  full  data  for  the  building  is  not  available,  intelligent  defaults  are  used 
based  on  ASHRAE,  extensive  background  from  CBECS,  research  papers,  and  expert  systems 
developed  by  energy  modeling  professionals. 

REM  is  useful  in  developing  a  starting  point  in  understanding  how  the  studied  building  is 
operating  using  a  model  derived  from  a  large  set  of  existing  buildings  that  are  operating 
correctly.  Having  an  understanding  of  the  building  energy  sensitivities  and  how  building  energy 
use  differs  from  typical  buildings  allows  one  to  focus  the  energy  conservation  work;  evaluators 
can  look  at  their  portfolio  to  find  outliers;  or  users  can  use  prioritize  retrofit  budget  where  it  is 
needed  most. 

Several  inputs  to  the  energy  model  are  driven  by  observations  from  satellite/aerial  imagery  and 
survey  responses  from  building  managers.  Building  and  operational  attributes  of  a  particular 
building  not  properly  identified  can  impact  modeling  results.  This  is  not  a  limitation  with  REM, 
but  a  general  limitation  with  simulation  in  general. 

Accurate  modeling  of  building  systems  is  an  important  factor  in  developing  useful  energy 
models.  The  downside  to  focusing  on  these  building  systems  and  their  operation  is  that  they  add 
a  high  level  of  detail  to  a  process  whose  goal  is  to  remain  rapid  and  agile.  Engineers  and  energy 
analysts  who  want  to  do  more  detailed  analyses  can  move  REM  data  to  eQuest  or  EnergyPlus  for 
detailed  work  in  those  tools,  which  may  require  more  expertise  and  detailed  inputs.  Constructing 
the  initial  model  using  REM  can  yield  substantial  timesavings  versus  initial  model  creation  in 
eQuest  or  Energy  Plus  tools  (Schneider,  2011). 


6 


3.0  PERFORMANCE  OBJECTIVES 


Table  1.  Performance  objectives. 


Performance 

Objective 

Metric 

Data 

Requirements 

Success  Criteria 

Results 

Quantitative  Performance  Objectives 

Correlation  of 
REM  with 
annual  energy 
electricity  and 
fiiel  intensity 

kWh  and 
therms 

Utility  history 
and/or  energy 
meter  data 
(compared  to 
gbXML  model 
data) 

Annual  electric  and 
natural  gas  energy 
+/-10%  compared  to 
baseline  historical 
utility  data 

Annual  electric  and 
natural  gas  within 
good  to  reasonable 
prediction  levels  as 
defined  in  literature. 

•  Results  were  within  1 0%  error  on 

7  out  of  25  buildings  for  electric. 
Two  buildings  fell  within  +/-10% 
for  natural  gas.  Overall,  there  was 
81.88%  average  accuracy  for 
electric  (18.12%  MAPE). 

•  Models  for  electric  use  in  office 
buildings  performed  better  than 
models  for  barracks  or  specialty 
use  buildings,  with  85.70% 
average  accuracy  (14.30% 

MAPE).  Accuracy  for  natural  gas 
averaged  58.2%  (41.80%  MAPE). 

•  Principle  reasons  for  deviations 
could  be  related  to  flawed  meter 
data,  weather  anomalies, 
occupancy  variations  in  building 
usage,  or  interior  space  utilization 
differences  (see  8.0). 

•  Also,  deviations  may  point  to 
operational  inefficiencies  that 
should  be  addressed  through  re¬ 
commissioning  for  energy  and 
cost  savings. 

•  Although  not  within  1 0%  error, 
electric  MAPE  values  were  within 
the  1 1  -20%  threshold,  considered 
“good.”  Forecasts  of  natural  gas 
usage  at  41.80%  MAPE  were 
within  21%-50%,  considered 
reasonable  (Lewis,  1982;  Chen  et 
al.,  2008) 

Correlation  of 
REM  with 
overall  annual 
EUI 

kBtu/ft2 

Utility  history 
and/or  energy 
meter  data 
(compared  to 
gbXML  model 
data) 

Annual  Energy 

Intensity  +/- 25% 
compared  to  baseline 
historical  utility  data 

Annual  EUI 
predictions  within 
good  to  reasonable 
levels  as  defined  in 
literature. 

•  14  out  of  25  buildings  were  within 
+/-  25%  MBE  in  predicting 
overall  EUI.  Average  accuracy 
was  77.56%  (MAPE  of  22.44%) 

•  MAPE  results  fall  within  the  2 1  %- 
50%  threshold,  considered 
reasonable.  As  above,  deviations 
may  be  related  to  inaccurate  meter 
data,  operational  inefficiencies, 
weather  anomalies,  or  space 
utilization. 

kWh  =  kilowatt  hours  kBtu  =  thousand  British  thermal  units 

MAPE  =  mean  absolute  percentage  error  ft2  =  square  feet 


MBE  =  mean  bias  error 
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Table  1.  Performance  objectives  (continued). 


Performance 

Objective 

Metric 

Data 

Requirements 

Success  Criteria 

Results 

Quantitative  Performance  Objectives  (continued) 

Variance  in 
monthly 
consumption 
(billing  history) 

% 

Utility  rates, 
energy  meter  data 
and  modeled 
energy  data  for 
each  building 

Acceptable  values  are  a 
coefficient  of  variation 
(CoV)  of  the  root  mean 
squared  error 
(CVRMSE)  of  <_15%. 

•  Results  were  within  15%  CVRMSE  for 
a  total  of  three  buildings  using  billing 
history  and  cost  as  metrics.  An 
additional  two  buildings  were  within 

20%  CVRMSE. 

•  Additional  simulation  runs  did  not 
attempt  to  tune  the  modeled  results  to 
match  metered  values,  but  the  CVRMSE 
provides  a  snapshot  of  how  baseline 
models  aligned  with  metered  data. 

•  It  was  not  anticipated  that  initial  models 
would  align  within  15%,  as  this  is  the 
standard  that  calibrated  models  are 
working  towards  and  is  outside  of  REM 
intent.  Buildings  with  the  closest 
calibration  were  selected  for  exploration 
of  design  alternatives  for  ECMs. 

Testing  the 

REM  process 
for  design 
alternatives  to 
model  PES 

%  energy 
savings  in 
kWh  and 
therms 

gbXML  file  and 
GBS  design  files 

Design  strategies  will 
attempt  to  achieve  energy 
savings  greater  than  30% 

•  ECMs  explored  basic  and  advanced 
design  strategies  for  five  buildings. 
Savings  greater  than  30%  was  achieved 
on  three  out  of  the  five  buildings. 

•  The  two  buildings  that  did  not  achieve 
the  target  already  had  undergone  energy 
retrofits,  which  were  reflected  in  the 
models. 

Qualitative  Performance  Objectives 

Ease  of 
learning 
technology  and 
expertise 
required  to 
create  REM 
models 

Person  hours 
of  training  to 
complete 
building 
models 

Training 

Curriculum;  Hours 
required  for 
successful 
completion  of 

REM  training 
program. 

On  average  <  6  days  to 
learn  technology  and 
complete  1  st  building 
model  and  generate  an 
energy  report.  After 
successful  completion  of 
first  REM  on  average  <  2 
days  per  building  to 
complete  models  and 
generate  reports. 

The  1  year  of  technology  transition  has  not 
yet  passed,  however  preliminary  results 
indicate  that  energy  models  can  be 
completed  in  less  than  3  hours  after  the 
process  is  learned. 

User 

Satisfaction 

Satisfaction 
with  REM 
workflow 
and 

processes 

Responses  from 
informal  interviews 
and  anecdotal 
observations 

Users  are  generally 
satisfied  with  the  REM 
process,  tools,  and  results 

Participants  indicate  a  high  level  of 
satisfaction  with  the  workflow. 

Ability  to  scale 
process  across 
the  DoD 

Number  of 
REM  trained 
personnel  at 
end  of  pilot 
study 

Participants  active 
in  training  program 
and  completion  of 
training 

Five  individuals  trained 
and  independently 
creating  REM  models  at 
completion  of  first  year 
of  technology  transition. 

At  this  point  in  time,  three  individuals  have 
received  training,  with  others  scheduled  for 
training  in  the  future. 

CoV  =  coefficient  of  variation 

CVRMSE  =  Coefficient  of  Variation  of  the  Root  Mean  Square  Error 
PES  =  potential  energy  savings 
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4.0  FACILITY/SITE  DESCRIPTION 


4.1  FACILITY/SITE  LOCATION  AND  OPERATIONS 

Researchers  visited  a  total  of  10  installations  across  six  climate  zones  (Figure  2)  between 
December  2012  -  March  2013,  and  selected  23  buildings  for  inclusion  in  the  core  analysis  of  the 
study. 


Fort  Leonard 
Wood,  MO  (5) 


JBLM,  WA  (5) 


Peterson 
AFB,  CO  (4) 


Port  Hueneme,  CA  (3) 


ERDC-CERL, 

IL  (3) 

Naval  Station  Great 


Seymour  AFB,  NC  (5) 


Yard,  ME  (1) 


Naval  Weapons  Sta. 
EARLE,  NJ  (2-3) 


Figure  2.  Site  locations. 

4.2  FACILITY/SITE  CONDITIONS 

Prior  to  scheduling  the  site  visits,  researchers  received  verification  that  meter  data  was  complete 
and  usable  by  Engineer  Research  and  Development  Center,  Construction  Engineering  Research 
Laboratory  (ERDC-CERL),  a  partner  on  this  project  working  via  a  Cooperative  Research  and 
Development  Agreement  (CRADA)  set  up  for  this  project.  Site  visits  included  engagement  with 
the  installation  point  of  contact  (POC)  and  review  of  the  completed  energy  questionnaire.  The 
research  did  not  interfere  with  ongoing  operations. 

Table  2.  Site  information. 


Installation 

Buildings 

Installation 

Buildings 

ERDC-CERL 

3  Offices 

Port  Hueneme 

4  Offices  (3  excluded) 

Fort  Leonard 

Wood 

1  Office;  3  Barracks 
(1  excluded  from 
analysis),  1  Gym 

Portsmouth 

1  Barracks 

Joint  Base  Lewis 
McChord 

2  Offices  (1  excluded); 

2  Barracks  (1  excluded) 

Seymour  AFB 

1  Office;  1  Cafeteria;  1  School;  1  Fire 
station;  1  Automotive  Facility 

Panama  City 

2  Offices;  1  Barracks 

Earle  Naval  Weapons 
Station 

1  Office;  1  Auto  Facility;  1  Cafeteria 
(all  excluded) 

Peterson  AFB 

4  Offices 

Great  Lakes 

2  Barracks;  1  Drill  Hall  (all  excluded) 

AFB  =  Air  Force  Base 
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5.0 


TEST  DESIGN 


5.1  CONCEPTUAL  TEST  DESIGN 

This  project  evaluated  technical  performance  and  cost  characteristics  of  estimating  energy 
consumption  of  buildings  by  conducting  REM  simulations.  These  simulations  were  then 
compared  blindly  to  historical  energy  use  information  of  the  same  studied  buildings.  A  subset  of 
five  buildings  was  further  processed  with  the  design  alternatives  capabilities  of  REM  software 
tools  in  order  to  estimate  how  much  energy  could  be  saved  by  applying  ECMs.  Design 
alternatives  were  selected  for  each  of  the  five  buildings  by  the  project  team  and  simulation 
estimates  are  included  in  this  report.  The  test  phases  are  described  in  Table  3. 

Table  3.  Test  phases. 


Test  Phase 

Activity 

Reality  capture 

Identification  of  35  candidate  facilities  of  various  types  in  different  locations;  reduced 
the  number  to  23  buildings  for  aggregate  analysis  because  of  meter  data  quality 
concerns.  Energy  questionnaire  collection.  Meter  data  validation  by  ERDC-CERL. 
Candidate  buildings  identified  on  satellite  and  verified  by  installation  POCs.  Site  visit 
for  reference  photos  and  clarification  on  questionnaire  responses. 

Model 

Used  Formlt  conceptual  modeler  in  the  field  to  create  3D  building  model,  refined  the 
model  in  Revit  based  on  energy  survey,  site  observations,  and  reference  photos.  Vasari 
workflow  explored  a  remote  approach  using  software-integrated  satellite  imagery  and 
the  energy  survey. 

Generated  energy  models  based  on  conceptual  model  building  location,  geometry, 
energy  settings,  ASHRAE  defaults  where  energy  settings  were  not  provided,  and 
weather  information.  Performed  Conceptual  Energy  Analysis  driven  by  GB S/DOE  2.2 
engine.  Produced  energy  reports. 

Analyze 

Compared  modeled  results  to  actual  utility  meter  data  and  to  benchmarking  results 
using  CBECS.  Compared  REM  to  time  and  cost  of  audits.  Reviewed  the  energy 
analysis  findings  under  the  High  Performance  and  Sustainable  Building  Guiding 
Principles  Compliance  Pathways  for  building  efficiency  and  sustainability  goals  for 
CVRMSE,  using  billing  rates.  Five  of  the  study’s  23  buildings  that  were  within  an 
acceptable  tolerance  of  CVRMSE  calibration  were  further  processed  with  the  design 
alternatives  capabilities  of  GBS,  informed  by  PES  analysis  across  a  range  of  building 
parameters. 

Technology  transfer  & 
reporting 

Workshop,  Webinar  and  Curriculum  Development.  Report  development  and 
submission. 

5.2  BASELINE  CHARACTERIZATION 

The  historical  metering  data  was  used  as  a  reference  condition  to  determine  the  technical 
performance  accuracy  of  the  REM  method  and  the  existence  of  historical  natural  gas  and  electric 
metering  information  was  a  prerequisite  for  a  building  to  participate  in  this  study.  ERDC-CERL 
requested  building  natural  gas  and  electric  meter  data  at  the  most  granular  level  available  from 
candidate  installations.  CERL  then  conducted  a  review  of  this  data  to  ensure  that,  at  minimum, 
there  were  12  months  of  reliable  natural  gas  and  electric  meter  data  for  each  building. 
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The  inputs  for  the  energy  model  were  derived  using  imagery  and  responses  to  the  site  survey, 
and  focused  on  rapid  baseline  characterization  of  the  building  geometry,  operations  and  systems. 
The  REM  workflow  also  does  not  utilize  floorplans  or  model  interior  walls,  opting  instead  for 
ASHRAE  standard  perimeter-core  space  simplification  and  a  maximum  width  of  perimeter  zone 
to  minimize  the  error  introduced  by  removing  interior  partitions.  The  REM  models  also  do  not 
designate  different  space  utilizations  within  a  building,  so  buildings  with  different  space 
utilizations  (i.e.,  office  and  lab)  are  modeled  as  one  building  type  per  generalizations  similar  to 
the  building-wide  defaults  recommended  in  ASHRAE  90.1  vs.  the  space-by-space  method. 
Accurate  modeling  of  interior  spaces  is  possible  with  the  software  tools,  however  this  requires  a 
significant  time  investment  to  collect,  organize,  and  translate  building  plans  into  the  model,  and 
would  require  additional  expertise  from  DoD  end  users  that  is  not  of  sufficient  value  for  the 
purposes  of  a  REM  survey.  Similarly,  building  schedules  may  not  be  uniform  throughout  the 
building,  or  consistent  on  a  weekly,  monthly  or  annual  basis.  Researchers  used  information 
provided  by  installation  staff  to  determine  schedule  selection  in  the  modeling  and  energy 
analysis  tools.  Several  installations  provided  monthly  totals  instead  of  interval  meter  data  as 
requested,  thus  in  these  cases,  few  insights  regarding  accuracy  of  schedule  assumptions  could  be 
gleaned.  It  was  assumed  that  weather  for  the  year  of  meter  data  submitted  was  not  anomalous. 

Some  installations  submitted  monthly  interval  data,  while  others  submitted  15  minute  interval 
data.  Several  buildings  were  eliminated  during  this  validation  stage,  due  to  apparent  issues  with 
the  meter  data.  Other  datasets  were  validated  and  included  in  the  study,  only  to  have  the  meter 
data  later  determined  to  be  unreliable  when  released  from  ERDC-CERL  to  Autodesk  for 
comparison  with  REM  results.  Several  buildings  included  in  the  study  have  meter  data 
anomalies,  such  as  large  spikes  in  usage  that  may  or  may  not  be  accurate  (see  Appendix  C). 
Additionally,  modeled  energy  results,  and  metered  data  were  compared  to  the  DOE  Index  for 
Commercial  Buildings,  which  utilizes  data  from  the  Energy  Information  Administration  (EIA) 
2003  CBECS  using  the  Building  Energy  Data  Book  tool.  Primary  search  criteria  were  climate 
zone  and  building  type,  followed  by  size  and  vintage,  if  sample  sizes  were  sufficient  (n>10)  to 
allow  further  refinement. 

5.3  DESIGN  AND  LAYOUT  OF  TECHNOLOGY  COMPONENTS 

Researchers  explored  the  various  software  tools  and  workflows  to  better  assess  capabilities  and 
then  optimize  scalability  for  DoD  when  technology  is  transferred  (see  section  2.1).  Some  of  the 
tools  have  overlap  in  terms  of  their  functional  attributes,  and  the  portability  of  file  formats 
between  tools  allows  users  a  great  deal  flexibility  in  determining  a  workflow  (Figure  3) 
depending  on  the  level  of  detail  desired,  expertise,  and  time  constraints  (See  Section  6.5  for 
discussion  of  attributes  of  workflows.) 
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In  the  REM  workflow,  the  mass  form  geometry  is  created  using  satellite  imagery.  Mass  floors  are  then  created  to 
reflect  the  number  of  levels  and  floor  to  floor  heights  of  the  building,  which  are  informed  by  the  questionnaire 
responses  and  satellite  images  (See  Section  5.1).  Energy  settings  are  then  selected  based  on  questionnaire 
information  or  satellite  information  in  the  case  where  supplied  information  is  inadequate.  The  energy  model  is  then 
enabled  and  zoning  is  created  based  on  ASHRAE.  The  energy  model  report  is  then  generated  in  GBS.  Enhanced 
analysis  is  then  possible  with  GBS. 


Figure  3.  Visual  depiction  of  DoD  technical  workflow. 

5.4  OPERATIONAL  TESTING 

The  relevant  mode  of  operation  is  a  standard  methodology  outlined  (Table  4).  The  testing 
occurred  between  October  2012  and  October  2013  (see  Figure  13  Gantt  chart  in  the  ESTCP  Final 
Report). 
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5.5  SAMPLING  PROTOCOL 


Table  4.  Sampling  Parameters  and  Types. 


Performance  Objective 

Parameters 

Number  and  Type  of  Samples 

Correlation  of  REM  with  annual 
energy  electricity  intensity, 
annual  fuel  intensity,  and  overall 
EUI 

Electricity  and  natural  gas 
data  from  model  and  from 

meters 

Minimum  of  1  year  of  meter  data  information  plus 
model  data.  Produced  monthly  and  annual  graphs. 
CBECS  data  included  annual  kBtu/  ft2  electric,  natural 
gas,  and  EUI,  respectively. 

Variance  in  monthly 
consumption  (billing  history) 

Billing  rates 

Utility  bills  were  unavailable.  CVRMSE  was 
calculated  between  monthly  modeled  versus  metered 
billing  costs  based  on  utility  rates  provided  and  using 
monthly  energy  use. 

Energy  reduction  through  GBS 
modeling  of  ECMs 

%  energy  savings  in  kWh 
and  therms;  cost  savings  in  $ 

PES  analyses  within  GBS  were  used  to  identify 
design  alternatives  for  ECMs  for  five  buildings. 
Documentation  of  energy  and  cost  savings  vs.  model 
and  meter  data. 

Time  and  cost  to  energy  model 

Hours  or  hours/ft2;  $/  ft2 

Published  data  in  preparation  for  publication  was  used 
to  assess  average  time  and  cost  requirements  for 
ASHRAE  Level  2  audits. 

Ease  of  learning  technology  and 
expertise  required,  satisfaction, 
scalability 

Hours  of  training  to  complete 
initial  building  energy 
models.  Satisfaction  with 

REM  workflow  and 
processes. 

Autodesk  staff  to  optimize  workflows  based  on 
experience,  input  requirements  and  constraints, 
conduct  training  with  DoD  personnel,  gauge 
satisfaction  and  time  required  to  learn  workflow. 

5.6  SAMPLING  RESULTS 

Table  5.  Summary  info  on  data  collected. 


Division 

#  of 

Buildings 

Visited 

#  of 

Buildings 
in  Core 
Study  Set 

Captured  Data 
Pre-site  Visit 
for  each  Building 

On  Site  Information 
for  each  building 

Models  for  each 
building 

Reports  1 

for  each  building 

Army 

8 

7 

•  Meter  data 

•  Energy  survey 

•  Location  and 
satellite  image 

•  Photos  of  building 
exterior  (often  non- 
essential).  Reference 
measurement  of 
building  footprints 
(non-essential) 

•  Clarifying  questions 
re:  energy  survey 

•  Conceptual 

3D  models 

•  GBS XML 

GBS  Dashboard 
Charts  and  data 
tables. 

Monthly  and 
annual  tables  and 
graphs  of  modeled 
results  plotted  in 
relation  to  results 
from  building 
meters. 

Navy 

14 

5 

Air  Force 

9 

9 

Joint 

4 

2 

Total 

35 

23 

(12 

removed 
due  to  data 
issues) 
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6.0  PERFORMANCE  ASSESSMENT 


Data  collected  during  the  demonstration  provides  information  necessary  to  assess  the 
effectiveness  of  REM  relative  to  the  performance  objectives  defined  in  Table  1.  The  following  is 
a  summary  of  the  analysis  in  support  of  the  performance  objectives. 

6.1  OVERALL  CORRELATION  OF  MODELED  RESULTS  TO  METER  DATA 

These  measures  quantified  the  effectiveness  of  REM  to  estimate  natural  gas,  electric  and  overall 
energy  usage  of  individual  buildings  within  10%  error  compared  to  meter  data  provided  by  the 
installation.  Prior  case  studies  that  guided  establishment  of  the  10%  error  targeted  traditional 
commercial  office  buildings  with  standard  operating  hours  and  usage.  DoD  buildings  in  the 
sample  vary  widely  in  their  occupancy  and  usage  and  a  re-established  success  criteria  of  <20% 
error  is  a  better  metric  to  evaluate  forecast  accuracy.  Forecast  performance  was  assessed  using 
MBE  and  MAPE.  Lewis’s  interpretation  of  MAPE  results  (1982)  is  criteria  used  to  judge  the 
accuracy  of  the  forecast  and  is  summarized  in  Table  6. 

Table  6.  Typical  MAPE  values  for  model  evaluation. 


MAPE (%) 

Evaluation 

MAPE  #  10% 

High  accuracy  forecasting 

10%  <  MAPE  #  20% 

Good  forecasting 

20%  <  MAPE  #  50% 

Reasonable  forecasting 

MAPE  >  50% 

Inaccurate  forecasting 

Source:  Lewis  (1982) 


Energy  use  is  a  frequently  tracked  metric  for  many  buildings,  yet  there  are  many  buildings  that 
do  not  have  meters  installed,  meters  are  not  functioning,  and  or  data  is  not  usable  (see  list  of 
meter-related  issues  in  Section  8.0).  REM  predicts  how  buildings  should  be  performing  (or 
where  buildings  are  potentially  used  in  non-standard  ways),  based  on  their  use  profile,  unique 
geometry,  generalized  use  schedules,  and  location  and  construction  characteristics  for  buildings 
of  their  type  and  region.  Where  model  input  parameters  are  not  known,  many  sources  are  used  to 
define  defaults  based  on  CBECS,  design  tables  within  ASHRAE/Illuminating  Engineering 
Society  of  North  America  (IESNA)  Standard  90.1-2004,  scientific  research  papers  and  modeling 
best  practices.  This  provides  a  rational  baseline  of  information  from  which  to  make  asset 
management  decisions. 

The  models  predicted  energy  usage  using  GBS,  driven  by  the  DOE  2.2  engine.  Meter  data 
received  from  the  installations  was  reviewed  by  a  third  party  (ERDC-CERL)  prior  to  comparison 
with  modeled  estimates.  In  some  cases,  despite  verification,  subsequent  issues  were  encountered 
with  the  meter  data  that  required  removal  from  the  study  or  aspects  of  the  analysis.  Of  a 
population  of  35  buildings,  a  total  of  23  buildings  were  included  in  core  analyses.  A  total  of  12 
buildings  were  excluded  from  core  analyses  due  to: 

•  Questionable  meter  data  and  scaling  issues  -  three  Earle  Naval  Weapons  Station 
buildings  and  one  Joint  Base  Lewis  McChord  building 
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•  Building  occupancy  concerns  -  one  Fort  Leonard  Wood  building  and  one  Joint  Base 
Lewis  McChord  building 

•  Absence  of  natural  gas  data  -  Port  Hueneme,  three  buildings  and  Naval  Station  Great 
Lakes,  three  buildings) 

Electric  Results 

Overall,  the  MAPE  for  electric  results  was  18.12%,  representing  average  accuracy  of  81.88% 
(n=23)  (Table7;  Appendix  B).  Although  MAPE  of  18.12%  is  outside  of  the  success  criteria 
described  in  the  performance  objectives,  stated  as  +/-10%,  it  is  still  considered  a  “good”  forecast 
according  to  criteria  established  by  Lewis  (1982).  Correlations  in  energy  use  curves  were  evident 
in  most  buildings. 

Natural  Gas  Results 

Natural  gas  results  for  the  23  analyzed  buildings  had  a  MAPE  of  41.80%,  or  an  absolute  average 
accuracy  of  58.20%  (Table  7).  This  is  outside  of  the  project’s  stated  success  criteria  of  +/-  10% 
error,  but  is  considered  to  be  within  the  criteria  of  a  “reasonable”  forecast  Lewis  (1982),  as  it  is 
within  the  range  of  21-50%.  In  general,  the  models  appear  to  be  less  accurate  in  predicting  actual 
natural  gas  usage  than  electric  usage  in  DoD  buildings.  This  may  be  due  to  errors  in  modeling 
results,  but  the  natural  gas  model  results  align  closer  with  CBECS  natural  gas  values  than  the 
metered  natural  gas  values  and  may  point  to  other  sources  of  error.  Natural  gas  is  much  more 
sensitive  to  heating,  ventilation  and  air  conditioning  (HVAC)  settings  and  climate  than  electricity 
because  natural  gas  in  the  energy  model  is  only  for: 

•  Hot  water  (very  small  amount  but  very  sensitive  to  user  operation) 

•  Heating  (very  sensitive  to  climate  and  building  operation);  feedback  from  personnel 
indicates  that  HVAC  systems  are  often  operated  excessively 

•  Reheat  (very  sensitive  to  HVAC  settings  and  often  set  up  very  poorly) 

•  Infiltration 

•  Various  very  large  process  loads  like  a  pool,  cafeteria,  or  other  unique  things  that  are  not 
typically  part  of  REM 

These  issues  can  be  checked  easily  in  buildings  and  are  good  candidates  for  re-commissioning. 
Overall,  metered  values  are  much  higher  than  modeled  values,  with  the  exception  of  a 
Leadership  in  Energy  and  Environmental  Design™  (LEED™)  building,  and  two  dorm  buildings 
with  questionable  occupancy  levels. 

Energy  Use  Intensity  Results 

EUI  results  had  a  MAPE  of  22.44%  (N=23),  representing  77.56%  absolute  average  accuracy  in 
EUI  predictions  (SD=13.48%).  This  MAPE  for  the  pooled  set  of  buildings  is  within  the  stated 
performance  objective  criteria  of  =/-  25%  error,  and  is  considered  “reasonable”  according  to 
established  criteria,  as  it  falls  between  21%-50%  MAPE  (Table  7;  Appendix  B).  The  highest 
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energy  use,  represented  by  EUI  (kBtu/ft2),  was  found  in  a  cafeteria,  dormitories  and  a 
gymnasium. 


Table  7.  Summary  data  for  all  analyzed  buildings  (n=23). 


Statistics 

Electric 

Natural  Gas 

EUI 

Average  Accuracy 

81.88% 

58.20% 

77.56% 

MAPE 

18.12% 

41.80% 

22.44% 

In  most  cases,  there  was  closer  alignment  of  simulation  data  to  CBECS  result,  and  researchers 
attribute  the  deviation  between  the  model  and  meter  data  to  buildings  that  are  performing  worse 
than  should  be  expected  based  on  their  attributes.  To  further  explore  the  results,  analyses  were 
clustered  by  building  use  type  and  plotted  against  benchmarking  results  from  the  CBECS  2003 
survey.  Examination  of  this  range  of  buildings  improved  the  findings  of  the  demonstration  by 
allowing  visibility  of  trends  within  use  categories.  The  various  building  types  included  13 
offices,  five  barracks,  five  special  use  buildings  (fire  station,  gym,  school,  auto  facility,  and 
cafeteria). 


Figure  4.  Comparison  of  meter  and  model  data  in  relative  kBtu/ft2. 
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EUI  (kBtu/SF)  for  DoD  Study  Buildings 


Figure  5.  Comparison  of  meter  and  model  data  in  relative  kBtu/ft2. 

6.1.1  Offices  Subset 

In  total,  13  offices  across  seven  Army,  Navy  and  Air  Force  locations  participated  in  the  core 
analysis  in  the  study.  The  offices  ranged  in  size  from  4,800  gross  square  foot  (GSF)  to  281,732 
GSF  (Appendix  B).  For  photos  of  buildings,  please  refer  to  the  ESTCP  Final  Report. 

Overall,  the  offices  averaged  85.70%  in  accuracy  when  comparing  modeled  estimates  for  electric 
data  to  electric  meter  data,  representing  average  MAPE  of  14.30%,  which  is  considered  a  good 
forecast  based  on  criteria  proposed  by  Lewis  (1982).  While  electric  modeling  results  for  offices 
aligned  closely  with  actual  metered  usage,  natural  gas  models  for  offices  were  on  average  only 
49.48%  accurate,  with  a  MAPE  of  50.52%,  which  is  considered  a  reasonable  forecast,  but  is  on 
the  cusp  of  being  considered  inaccurate  (MAPE  >51%). 

Energy  surveys  were  revisited  to  investigate  buildings  where  modeled  estimates  were  more  than 
20%  different  than  meter  data.  For  the  office  electric  data,  only  Naval  building  1100  in  Port 
Hueneme  (33.83%  error)  and  Office  Building  1345  at  Peterson  (-21.33%  error)  were  outside  of 
this  threshold  (see  the  ESTCP  Final  Report  for  discussion  of  these  buildings).  Only  three 
buildings  were  >80%  accurate  for  natural  gas  estimates  and  those  included  Port  Hueneme  1100 
and  the  two  smallest  buildings,  1345  and  1485  at  Peterson  AFB  (Figure  6).  The  office  buildings 
demonstrated  correlation  in  trend  profile  shape,  but  had  significant  amplitude  differences,  with 
the  building  consuming  more  natural  gas  than  predicted  by  the  model  (Appendix  C). 

In  all  cases,  with  the  exception  of  Building  1100,  building  natural  gas  meter  and  EUI  data  is 
higher  than  what  is  predicted  in  the  models.  While  the  possibility  exists  that  differences  could  be 
attributed  to  natural  gas  use  related  calculations  in  the  models,  it  should  be  noted  that  in  general, 
building  natural  gas  usage  and  EUI  were  also  significantly  higher  than  CBECS  values. 
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Researchers  attribute  the  deviation  between  the  model,  meter,  and  CBECS  results  to  buildings 
that  are  performing  worse  than  should  be  expected  based  on  their  location  and  attributes. 

Overall,  modeled  and  metered  electric  results  aligned  relatively  well  with  CBECS  2003 
benchmarking  results  for  office  building  (see  Table  10  and  Figure  23  in  the  ESTCP  Final 
Report).  MAPE  for  electric  meter  data  compared  to  CBECS  was  30.13%  (69.87%  average 
accuracy).  Natural  gas  meter  data  deviated  greatly  from  CBECS  with  MAPE  of  122.44%  (- 
22.44%  accuracy),  while  model  data  was  closer  aligned  to  CBECS  with  MAPE  =  42.05% 
(57.95%  accuracy). 

The  REM  workflow  seems  reasonably  accurate  for  estimating  overall  EUI  for  DoD  office 
buildings.  Overall,  of  13  offices  sampled,  the  MAPE  was  22.66%,  or  an  average  of  77.34% 
accurate.  Three  office  buildings  (350,  3369  and  1485)  were  within  90%  accuracy  and  an 
additional  three  (110,  CERL  2,  and  1345)  were  within  80%  accuracy.  With  the  exception  of 
Building  1100  in  Port  Hueneme,  all  other  office  buildings  had  EUI  meter  data  that  was  higher 
than  predicted  EUI  for  each  building. 

Conclusion 

Overall,  the  REM  workflow  appears  to  be  a  good  method  for  predicting  electric  usage  and  a 
reasonable  method  for  EUI  predictions  for  DoD  office  buildings  when  looking  at  MAPEs  for  the 
pooled  set  of  office  buildings.  The  high  variability  in  natural  gas  results  for  individual  buildings 
and  overall  MAPE  for  the  pooled  set  of  office  buildings  needs  further  investigation.  DoD  office 
buildings  are  consuming  significantly  more  natural  gas  and  have  higher  EUI  values  than 
predicted  by  the  models  and  compared  to  similar  buildings  in  the  CBECS  database. 

Energy  use  curves  and  trend  profiles  provide  insight  on  seasonal  variations,  deviations  and 
correlations  for  the  buildings.  Deviations  are  likely  attributed  to  faulty  meter  readings,  weather 
anomalies,  or  operational  and  mechanical  issues  at  the  individual  building  level.  Next  steps 
should  include  working  with  individual  building  managers  to  investigate  operations,  system 
configurations  and  settings  and  to  attempt  to  elucidate  understanding  around  spikes  in  usage  or 
other  anomalies  in  the  meter  data.  In  some  cases,  there  may  be  issues  with  the  meter  data  itself 
and  there  is  also  a  possibility  that  the  meter  data  was  submitted  for  a  weather  year  that  was 
atypical. 
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Figure  6.  Percent  deviation  -  model  vs.  metered  electric,  natural  gas  and  EUI  for  offices. 
6.1.2  Barracks  Subset 

Of  the  23  buildings  analyzed,  five  were  dormitories  ranging  in  size  from  25,349  GSF  to  96,130 
GSF.  There  were  an  additional  four  dormitories  sampled  that  were  not  included  in  core  analysis 
due  to  questionable  meter  data.  For  photos  of  buildings,  please  refer  to  the  ESTCP  Final  Report. 

Overall,  dormitory  electric  estimates  were  on  average  73.25%  accurate  when  compared  to  meter 
data  (MAPE  =  26.75%).  While  this  is  outside  of  the  +/-  10%  success  criteria  established  in 
performance  objectives,  it  is  considered  a  reasonable  forecast  according  to  Lewis  (1982) 
(Table  6).  Modeled  natural  gas  predictions  averaged  56.66%  accuracy,  including  an  outlier  of 
179%  error  at  Panama  City  Building  484,  where  the  model  predicted  much  higher  gas  usage  than 
was  evident  when  examining  meter  data.  With  this  outlier  removed,  natural  gas  accuracy 
averaged  73.93%  with  a  MAPE  of  26.07%  (Appendix  B). 

When  all  five  barracks  were  aggregated,  energy  model  predictions  were  on  average  70.12% 
accurate  with  a  MAPE  of  29.88%  for  EUI  predictions.  The  highest  accuracy  was  with  barracks 
831,  937,  and  9136,  which  were  all  >78%  accurate.  Dorms  484  and  373  were  assumed  to  be 
100%  occupied  throughout  the  year,  and  this  is  not  a  reasonable  assumption  for  these  particular 
buildings  upon  reviewing  the  meter  data.  Similarly,  barracks  that  were  excluded  from  analysis 
also  had  occupancy  concerns  that  were  even  more  dramatic. 

Comparisons  of  barracks  buildings  to  CBECS  data  is  summarized  in  Figure  30  in  the  ESTCP 
Final  Report.  CBECS  data  for  dorms  was  not  useful  for  comparisons  due  the  small  sample  of 
dorms  in  the  2003  CBECS  survey.  As  a  result,  CBECS  values  are  based  on  larger  criteria  of 
“lodging”  within  each  climate  zone  in  order  to  have  sample  sizes  >10  for  CBECS  values. 
Additionally,  since  the  2003  survey,  we  have  seen  an  explosion  in  the  use  of  personal  devices 
such  as  laptops  and  tablets,  associated  increases  in  plug  loads  would  not  have  been  observed  in 
2003. 
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Figure  7.  Percent  deviation  -  model  vs.  metered  electric,  natural  gas  and  EUI  for  barracks. 


Conclusion 


The  REM  workflow  is  a  reasonable  approach  to  predicting  electric,  natural  gas  and  EUI  in 
barracks  buildings  that  have  consistent  occupancy  throughout  the  year.  Variable  occupancy  can 
skew  the  data  significantly. 

If  barracks  buildings  are  going  to  be  utilized  in  REM  workflows,  users  should  understand  that 
the  energy  models  assume  100%  occupancy.  Reduced  occupancy  levels  can  be  varied  in  GBS 
(i.e.,  75%  occupancy,  50%  occupancy);  however  seasonality  of  reduced  occupancy  cannot  be 
accounted  for  in  the  building  energy  model.  Given  the  highly  variable  nature  of  DoD  barracks 
and  lack  of  available  information  on  occupancy  levels  through  the  year,  the  REM  workflow  for 
barracks  may  not  be  ideal,  unless  users  are  comfortable  with  the  assumptions  described  above. 

6.1.3  Specialty  Use  Buildings  Subset 

In  addition  to  offices  and  barracks,  researchers  sampled  five  specialty  use  buildings  including  a 
dining  cafeteria,  school,  fire  station,  automotive  facility,  and  a  gym.  All  buildings  were  under 
45,000  GSF  and  were  located  at  Seymour  AFB,  with  the  exception  of  one  gym  in  Fort  Leonard 
Wood,  MO.  For  photos  of  buildings,  please  refer  to  the  ESTCP  Final  Report. 

Overall,  energy  models  for  these  aggregate  specialty  use  buildings  were  an  average  of  80.58% 
accurate  for  electricity  estimates,  with  a  MAPE  of  15.72%,  indicative  of  a  good  forecast.  Energy 
models  were  an  average  of  70.80%  accurate  for  natural  gas  predictions,  with  a  MAPE  of 
29.20%,  indicating  a  reasonable  forecast.  Overall,  specialty  use  building  energy  models  were  an 
average  of  85.58%  accurate  for  predicting  EUI  with  a  MAPE  of  14.42%,  signifying  a  good 
forecast. 


The  REM  workflow  was  a  good  methodology  for  forecasting  electric  and  EUI  for  specialty  use 
buildings  based  on  MAPE  values  between  11-20%.  Further,  there  were  close  correlations  in 
trend  profiles  in  monthly  data  charts.  The  workflow  was  reasonable  at  predicting  natural  gas 
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usage  in  specialty  use  buildings  as  indicated  by  MAPE  value  of  29.2%.  Given  the  specialized 
building  types  in  this  subset,  and  the  limited  number  of  building  types  in  the  CBECS  database, 
comparison  to  CBECS  is  likely  unreliable  for  these  buildings. 
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Figure  8.  Percent  Deviation  -  model  vs.  metered  electric,  natural  gas  and  EUI  for  specialty 

use  buildings. 

6.2  VARIANCE  IN  MONTHLY  CONSUMPTION  (BILLING  HISTORY) 

The  calibrated  simulation  approach  in  this  study  involves  use  of  the  GBS  program  and  DOE  2.2 
engine  to  model  energy  use  of  existing  buildings  in  pre-retrofit  conditions  and  then  checked 
against  actual  measured  values.  Researchers  compared  monthly  utility  costs  between  the 
simulated  energy  values  and  actual  (metered)  energy  values  using  utility  rates  provided  by 
installation  POCs.  Utility  rates  and  usage  were  used  as  a  proxy  for  utility  bills,  with  the 
assumption  that  tariffs  are  included  in  the  rates  provided.  Researchers  calculated  the  CVRMSE 
for  each  building.  Additionally,  for  comparison,  CVRMSE  was  calculated  using  energy 
consumption  as  opposed  to  energy  costs. 

Buildings  were  not  modeled  again,  refined  or  recalibrated  to  get  within  15%  of  CVRMSE,  rather 
the  data  is  presented  as  a  picture  of  how  baseline  models  performed  relative  to  measured  values 
and  can  allow  identification  of  which  buildings  are  closest  to  acceptable  calibration  thresholds 
guided  by  under  the  ASHRAE  Guideline  14-2002. 

Three  building  models  were  within  15%  of  CVRMSE  for  billing,  including  Building  1  at 
Peterson  AFB,  Building  470  at  Fort  Leonard  Wood,  and  CERL  Building  2.  Two  additional 
buildings  were  within  20%  CVRMSE  for  billing  data,  including  Peterson  AFB  Building  350 
(18.73%  CVRMSE)  and  Building  4601  at  Seymour  AFB  (18.84%  CVRMSE).  Only  one 
building,  4103  at  Seymour  AFB  was  within  20%  CVRMSE  for  energy  usage.  The  five  buildings 
within  20%  CVRMSE  for  billing  were  used  to  explore  ECMs  through  Design  Alternatives  in 
GBS. 
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6.3  TESTING  THE  REM  PROCESS  FOR  DESIGN  ALTERNATIVES  TO  MODEL 
POTENTIAL  ENERGY  SAVINGS 

A  subset  of  five  buildings  was  selected  based  upon  falling  within  20%  CVRMSE  for  monthly 
utility  costs  and  design  alternatives  for  energy  conservation  were  explored  for  these  buildings. 
PES  analyses  tables  and  charts  automatically  generated  in  GBS  guided  researchers  in  assessing 
ECMs. 

How  Potential  Energy  Savings  Analysis  Works  (see  example  in  Figure  9) 

1 .  GBS  receives  model,  which  contains  any  specific  building  feature  design  options  defined 
in  Revit  or  Vasari; 

2.  For  any  building  features  that  were  not  specified,  GBS  inserts  appropriate  default  values 
for  the  building  type  and  location  and  runs  an  energy  analysis; 

3.  GBS  then  generates  50  alternative  design  variations  in  the  cloud  with  multiple  options  for 
14  building  parameters; 

4.  GBS  then  spawns  50  servers  and  runs  all  of  these  alternative  models  at  the  same  time; 
and 

5.  The  results  of  the  51  simulations  are  displayed  in  the  PES  chart  with  the  center  line 
reflecting  the  initial  baseline  run. 


Estimated  Savings  from  ECMs  is  expressed  as: 

•  Metered  savings  =  Metered  baseline  (kWh,  therms,  kBtu,  cost)-ECMpost  (kWh,  therms, 
kBtu,  cost) 

•  Modeled  savings  =Modeled  baserun  (kWh,  therms,  kBtu,  cost)-ECMpost  (kWh,  therms, 
kBtu,  cost) 

6.3.1  Design  Alternatives  -  CERL  Building  2 

For  CERL  Building  2,  an  office  in  Champaign,  Illinois,  there  was  a  CVRMSE  value  of  12.6%. 
PES  analyses  indicated  that  the  highest  energy  savings  could  be  gained  by  modifying  wall 
construction,  HVAC  type,  and  plug  load  and  lighting  efficiency  (Figure  9). 
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Figure  9.  PES  chart  for  CERL  Building  2. 

A  basic  package  set  of  measures  explored  upgrading  the  HYAC  system  while  utilizing  the 
infrastructure  of  the  existing  four  pipe  fan  coil  system,  reducing  lighting  power  density  (LPD) 
and  equipment  power  density  (EPD)  by  10%  and  installing  occupancy  sensors  and  daylighting 
controls.  An  advanced  package  included  all  measures  in  the  basic  package,  and  also  included 
addition  of  insulation  to  the  massive  brick  walls,  which  was  the  number  one  recommended 
improvement  but  also  potentially  the  most  costly. 

The  basic  package  set  of  measures  yielded  energy  savings  of  27.02%  and  $20,723  cost  savings 
compared  to  metered  energy  usage,  and  8.75%  compared  to  modeled  usage  and  $9,636  in  costs 
savings  compared  to  the  modeled  baserun.  Adding  wall  insulation  under  the  Advanced  Package 
yielded  an  additional  26%  energy  savings  (53.25%  total  savings)  and  an  additional  $19,080 
($39,803  total)  in  annual  cost  savings  compared  to  the  metered  baseline.  Differences  seen 
between  modeled  and  metered  savings  are  due  to  energy  usage  differences  between  the  runs, 
even  though  they  fall  within  15%  CVRMSE  calibration  criteria. 

Peterson  Bl,  an  office  in  Colorado  Springs,  Colorado,  had  a  CVRMSE  of  10.39%.  A  basic 
package  guided  by  PES  Results  (Figure  10),  explored  measures  including  reducing  LPD  and 
EPD  by  10%  each  and  adding  occupancy  and  daylighting  sensors  and  controls. 
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Figure  10.  PES  chart  for  Building  1  -  Peterson  AFB. 

There  were  no  high  priority  envelope  measures  indicated  by  the  PES  Chart,  thus  an  advanced 
package  explored  basic  package  measures  and  also  included  a  HVAC  change  to  premium 
efficiency  Variable  Air  Volume  (VAV)  with  reheat. 

Lighting,  equipment  and  control  improvements  yielded  a  40%  improvement  in  EUI  over  metered 
baseline  values,  and  a  12.81  %  improvement  over  EUI  from  the  modeled  baserun.  Annual  cost 
savings  were  $105,000  from  the  metered  baseline,  and  $17,727  from  the  modeled  baserun.  This 
discrepancy  is  linked  to  differences  between  metered  and  modeled  energy  estimates,  despite 
CVRMSE  values  within  15%.  With  HVAC  improvements  added  to  the  improvements  identified, 
facility  owners  may  realize  an  additional  $28,828  in  annual  cost  savings  ($134,463  total)  and 
1.95%  improvement  in  EUI  (41.95%  total)  over  metered  values. 

Office  building  number  470  at  Fort  Leonard  Wood,  Missouri,  had  a  CVRMSE  value  of  14.47% 
between  simulated  and  metered  monthly  cost  data.  PES  analyses  indicated  that  greatest  savings 
may  be  gained  from  upgrading  HVAC,  upgrading  glazing,  altering  window  area  and  improving 
plug  load  efficiency  and  lighting  efficiency  (Figure  11). 
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Figure  11.  PES  Chart  for  Building  470-Fort  Leonard  Wood. 

It  was  determined  that  changing  window  area  was  impractical  and  that  was  removed  from 
consideration.  Researchers  thus  investigated  a  basic  package  that  included  10%  improvement  to 
equipment  and  lighting  efficiency,  daylighting  and  occupancy  controls  and  HVAC  equipment 
improvements.  Researchers  selected  an  11.3  EER  packaged  VAV  system,  which  offers  a  slight 
improvement  in  efficiency  over  the  current  system,  but  does  not  require  an  overhaul  of  the 
existing  HVAC  infrastructure.  Existing  windows  are  double-pane,  thus  upgrades  to  these 
windows  were  considered  as  an  Advanced  Package  item.  Given  that  two  identified 
improvements,  HVAC  and  window  glazing  have  been  upgraded  in  the  building  relatively 
recently,  these  were  assessed  iteratively  in  addition  to  basic  package  measures  as  moderate  and 
advanced  packages. 

Exploration  of  design  alternatives  indicated  that  the  basic  package  of  ECMs,  including  lighting 
efficiency,  equipment  efficiency  and  control  improvements  yielded  a  31.79%  improvement  in 
EUI  and  cost  of  $73,067  over  metered  baseline  values,  a  9.7  %  improvement  in  EUI  and  $13,763 
in  savings  from  the  modeled  baserun.  Adding  HVAC  improvements  decreased  EUI  by  an 
additional  6.24%  and  resulted  in  an  additional  $9,506  in  cost  savings  ($82,573  total)  above  the 
metered  baseline  basic  package  reductions.  The  combination  of  HVAC  improvements  and 
window  upgrades  beyond  the  basic  measures  yielded  a  total  45.84%  improvement  in  EUI  and  an 
estimated  total  annual  cost  savings  of  $90,420. 

Office  building  350  at  Peterson  AFB  had  a  CVRMSE  value  of  18.37%.  The  PES  chart  (Figure 
12)  reveal  that  the  biggest  savings  can  come  from  alterations  to  glazing  type,  plug  load 
efficiency,  window  area,  lighting  efficiency  and  HVAC  type. 
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Figure  12.  PES  chart  for  Building  350  -  Peterson  AFB. 

It  was  determined  that  changes  to  window  area  are  not  feasible  and  that  existing  windows  are 
double  pane  and  tinted,  although  not  the  latest  in  window  design.  Therefore  researchers  explored 
a  basic  package  of  measures  that  investigated  reduction  in  plug  load  and  LPD  by  10%  each  plus 
the  addition  of  occupancy  sensors.  The  PES  analysis  indicated  that  HVAC  upgrades  presented 
low  opportunity  for  energy  savings,  particularly  since  it  would  involve  under  floor  air 
distribution  and  significant  modifications  to  the  existing  infrastructure.  HVAC  upgrades  were 
therefore  not  explored  for  this  building. 

Exploration  of  basic  measures  within  GBS  Design  Alternatives  revealed  that  improving  LPD  by 
10  %  and  EPD  by  10%  plus  adding  occupancy  sensors  is  estimated  to  improve  EUI  by  3.64%, 
and  reduce  energy  costs  by  $7,906  annually  compared  to  metered  data  for  the  building. 
Improvements  compared  to  the  modeled  baserun  indicated  an  improvement  of  7.49%  for  EUI, 
and  annual  cost  savings  of  $18,666. 

Building  4601,  a  fire  station  at  Seymour  AFB,  North  Carolina,  had  a  CVRMSE  value  of 
18.84%.  The  PES  chart  indicated  that  the  highest  energy  savings  could  come  from  upgrades  to 
HVAC  type,  wall  construction,  glazing  type,  LPD,  EPD,  and  window  area  (Figure  13).  The 
HVAC  system  that  indicated  improvement  was  an  under  floor  air  distribution  system,  which  was 
not  considered  feasible  for  this  building. 
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Figure  13.  PES  chart  for  Building  4601  -  Seymour  AFB. 

A  basic  package  of  measures  explored  10%  improvement  to  lighting  efficiency  (LPD  10%  less 
than  base  run),  10%  improvement  to  equipment  efficiency  (EPD  10%  less  than  base  run),  and  the 
addition  of  occupancy  sensors  (though  they  showed  only  marginal  savings  on  the  PES  chart). 
Window  glazing  could  be  improved  from  dual  pane  (baseline)  to  reflective,  insulated,  low  E 
windows  in  a  design  alternative  (Figure  13).  Exploration  of  basic  measures  within  GBS  Design 
Alternatives  helped  improve  EEH  by  8.40%  against  the  modeled  baserun  and  20.11%  compared 
to  the  metered  baseline.  Advanced  package  upgrades  yielded  a  modeled  savings  of  14.88%  and 
25.77%  against  the  metered  baseline. 

6.4  TIME  AND  COST  TO  ENERGY  MODEL 

While  constructing  BIM  models,  researchers  documented  the  time  required  for  model  creation 
and  energy  analysis  and  compared  time  required  for  each  workflow.  These  time-based  tests  were 
completed  after  a  significant  period  of  testing  and  workflow  refinement.  The  Revit-based 
workflow  required  an  average  of  27.54  minutes  (SD=6.75).  This  included  time  required  for 
creation  of  Formlt  mass  models  in  the  field,  model  enhancements  in  Revit,  and  energy  analysis 
in  Revit.  The  Vasari  based  workflow  required  an  average  of  17.81  minutes  (SD=5.87),  including 
integrated  energy  analysis  natively  in  Vasari.  Buildings  that  required  more  time  were  often  more 
complex  in  shape,  with  open  courtyards  or  drill  decks,  such  as  dormitory  building  484  in  Panama 
City,  FL. 

Given  the  data  above,  it  can  be  reasonably  assumed  that  conceptual  energy  models  and  analysis 
can  be  executed  in  1  hour  or  less  in  most  cases.  It  should  be  noted  that  this  assumes  that  no  travel 
is  required,  and  does  not  include  time  required  for  installation  personnel  to  answer  questions 
regarding  building  construction  and  operations  (the  average  time  required  for  this  aspect  is  2 
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hours  unless  assumptions  are  derived  from  satellite  photos  and  building  knowledge  instead  of 
building  documentation). 

Existing  methods  employed  by  the  DoD  to  measure  energy  consumption  and  building 
performance  has  historically  been  limited  to  benchmarking  or  energy  audits.  While 
benchmarking  methods  (such  as  Energy  Star®  Portfolio  Manager  or  CBECS)  are  quick,  they  do 
not  identify  specific  opportunities  for  energy-saving  occasions  in  buildings  and  are  often 
imprecise  because  they  are  not  customized  to  the  building  and  are  thus  prone  to  significant  error. 
They  too  are  subject  to  the  problems  with  access  to  and  availability  of  quality  data.  Traditional 
energy  audits  might  be  more  accurate  and  customized  than  benchmarking,  but  are  labor 
intensive,  expensive,  time-consuming,  and  require  a  high  level  of  expertise;  therefore,  they  are 
not  scalable  across  the  DoD  portfolio. 

Three  levels  of  energy  audits  are  typically  used:  walkthrough  (ASHRAE  Level  1),  general 
(ASHRAE  Level  2),  and  investment  grade  (ASHRAE  Level  3).  Requirements  for  each  of  these 
levels  can  often  lack  detail  and  it  is  generally  acknowledged  that  the  levels  do  not  have  distinct 
boundaries  (Shapiro,  2009).  Researchers  investigated  whether  Rapid  Energy  would  allow 
buildings  to  be  evaluated  within  a  shorter  time  and  smaller  budget  than  audits.  While  REM 
processes  include  the  many  of  the  benefits  of  Levels  1-3  energy  audits,  the  workflow  does  not 
provide  a  direct  match  to  one  particular  audit  type,  but  is  a  closer  match  to  the  outputs  of  an 
ASHRAE  Level  2  audit,  with  added  benefits  of  computer  simulation  of  a  Level  3  audit.  Results 
are  summarized  in  section  7.3 

Researchers  are  not  recommending  replacement  of  ASHRAE  audits  for  DoD  facilities.  However 
given  the  time,  expense,  and  expertise  required  for  ASHRAE  audits,  REM  approaches  can  be 
used  at  early  stages  of  energy  analysis  to  determine  buildings  that  are:  poorly  performing,  the 
best  candidates  for  retrofits,  and  may  present  the  best  potential  opportunities  for  energy  savings, 
with  the  added  benefit  of  computer  simulation  and  modeled  comparison  of  energy  conservation 
measures. 

6.5  QUALITATIVE  PERFORMANCE  OBJECTIVES 

Autodesk  has  recently  begun  technology  transfer  associated  with  the  ESTCP  Rapid  Energy 
Modeling  Demonstration  Project.  Researchers  have  assembled  a  training  curriculum  that 
includes  a  webinar,  hands-on  demonstration,  and  free  optional  enrollment  in  an  advanced 
certificate  program. 

Given  the  diversity  of  user  profiles,  goals,  availability  of  building  information  and  time,  there  are 
multiple  workflows  to  select  from,  based  on  end  goals,  systems,  access  and  user  profiles.  Options 
include  tablet-based  mobile  app  Formlt  for  conceptual  modeling  with  further  refinement 
available  in  Revit  or  Vasari.  Modeling  can  also  be  done  in  Revit  (detailed  and  conceptual 
modeling)  and  Vasari  (conceptual  modeling),  with  integrated  energy  analysis  through  GBS. 
Additional  refinement  of  the  energy  models  and  exploration  of  design  alternatives  can  be  done 
through  the  GBS  web  service. 

As  added  value,  DoD  staff  who  participated  in  the  ESTCP  demonstration  have  been  invited  to 
enroll  in  the  Autodesk  Building  Performance  Analysis  (BPA)  Certificate  Program  under  a  group 
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specifically  for  DoD  installation  staff.  The  personnel  who  have  received  training  to  date  have 
expressed  a  high  degree  of  satisfaction  with  results  from  ESTCP  for  their  installation,  the  ease 
and  speed  of  the  workflow,  and  content  in  reports  and  GBS  dashboards. 

There  is  intrigue  around  the  ability  to  produce  rapid  coarse  models  with  acceptable  results  that 
will  allow  staff  to  do  comparative  runs  and  quickly  answer  questions  about  which  parameter  has 
the  most  influence  on  the  building,  such  as  the  roof,  walls,  windows,  etc.  Staff  have 
communicated  that  REM  is  a  good  way  to  answer  questions  that  are  often  explored  by  DoD 
energy  managers  when  initially  contemplating  upgrades. 
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7.0  COST  ASSESSMENT 


Cost  estimates  are  organized  into  high  and  low  range  estimates  for  a  given  site.  The  high  range 
estimates  assume  computer  hardware  and  would  be  purchased.  This  assumed  hardware  purchase 
is  not  mandatory  as  existing  installation  laptop/desktop  computers  can  be  used  for  the  REM 
solution.  The  high  range  estimate  also  assumes  software  will  be  purchased  specifically  for  this 
task.  This  may  not  be  necessary  as  the  Air  Force,  Navy  and  the  U.S.Army  Corps  of  Engineers 
(USACE)  have  existing  enterprise  software  licenses  for  most  software  titles  used  for  REM.  The 
software  titles  not  covered  in  the  enterprise  agreements  are  available  as  free  software  downloads. 
The  low  range  cost  estimates  assume  the  utilization  of  existing  installation  hardware  and  the  use 
of  the  existing  enterprise  licensed  (or  free)  software.  The  operational  costs  after  the  up-front 
expenditures  would  be  $300  per  building  modeled  with  this  process  multiplied  by  the  number  of 
buildings  studied  at  an  installation.  Our  enterprise  cost  model  is  assuming  one  set  of  REM  tools 
per  installation  at  185  installations  as  a  full  deployment  of  this  technology  (Table  8). 

7.1  COST  MODEL 


Table  8.  REM  cost  model. 


Cost  Elements 
(Unit  One) 

Data  Tracked  During  the 
Demonstration 

Estimated  Costs  for 
Installation 

Unit 

Measure 

Hardware  capital  costs 
(Determined  based  on 
existing  desktop  or  laptop 
hardware) 

Typical  laptop/desktop  cost  during 
demonstration  period 

$041,350 

Per  user 

Software  costs 
(Determined  based  on 
access  to  DoD  Enterprise 
Software  Licenses 

Commercial-off-the-shelf  (COTS)  software 
fees  during  project  demonstration 

$044,590 

Per 

Installation 

Software  installation 

costs 

Time/labor  to  install/downloaded  software 

$200 

Per  user 

Operational  costs 

Level  of  effort  to  model  building,  add 
operational  attributes  and  produce  energy 
model  and  reports 

$300 

Per  building 

Software  maintenance 

Frequency  of  available  software  upgrades 

$600 

Yearly 

Operator  training 

Length  of  time  for  training  session  (lday) 

$500 

Per  user 

Upfront  set  up  costs  (not  including  operational  costs) 

$1300  (low  range)- 
$7240  high  range) 

COTS  =  Commercial-off-the-Shelf 


7.2  COST  DRIVERS 

The  hardware  costs  and  the  software  costs  for  this  project  are  well  known  and  reasonably 
predictable.  The  major  costs  have  been  variable  costs  -  of  collecting  the  operational  attributes  of 
the  buildings,  modeling  and  analyzing  buildings  and  creating  ECMs.  This  may  well  prove  to  be 
the  case  with  a  larger  population  of  sites.  Some  installations  have  this  operational  information 
readily  available;  at  other  locations  this  information  is  in  disparate  sources  that  make  collecting 
the  information  more  challenging  and  potentially  increasing  costs.  In  these  cases  were  this 
information  cannot  be  obtained  cost  effectively  (or  at  all),  default  ASHRAE  settings  and 
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program  defaults  are  used  to  cap  data  collection  costs.  An  additional  cost  driver  is  the  number  of 
buildings  studied  with  the  REM  methodology.  This  is  a  linear  progression  with  the  per-building 
REM  modeling  costs  outlined  in  section  7.1. 

Site  specific  and  regional  issues  may  come  into  play  through  the  interaction  of  the  installation 
POC  with  the  Information  Technology  (IT)  department  at  each  installation.  The  installation  of 
new  software  titles  may  require  IT  participation.  The  process  to  add  new  software  titles  vary  per 
agency  and  per  location.  This  is  a  potential  cost  variable  to  consider  when  deploying  this 
technology.  DoD  Enterprise- wide  life  cycle  costs  for  REM  components  are  summarized  in  Table 
24  in  the  ESTCP  Final  Report. 

7.3  COST  ANALYSIS  AND  COMPARISON 

REM  utilizes  recently  available  digital  3D  modeling  technologies.  The  REM  approach  does  not 
conform  precisely  to  existing  energy  assessment  methods  making  direct  comparisons  challenging 
but  in  the  end  productive.  In  addition,  as  there  is  no  energy  efficiency  equipment  installed  at  the 
installation  with  this  demonstration  project,  some  of  the  life  cycle  cost  methods  are  difficult  to  fit 
to  this  project. 

A  useful  approach  in  forming  a  life  cycle  cost  understanding  of  this  technology  is  to  compare 
REM  to  ASHRAE  Energy  Audits.  While  REM  and  energy  audits  approach  the  subject  matter 
from  different  viewpoints  with  substantive  differences  in  methodology,  there  is  a  significant 
overlap  in  the  data  produced,  accomplishing  similar  asset  management  objectives  and  in  the 
overall  desired  outcomes. 

REM  Per-Building  Operational  Costs  Compared  to  ASHRAE  Audits 

Reported  costs  for  detailed  energy  audits  may  vary  from  $0.12  up  to  $0,503  per  square  foot, 
depending  on  the  size  and  complexity  of  the  building  (Baechler  et  al.,  2011).  For  the  purposes  of 
this  study,  researchers  used  the  low-range  estimate.  In  this  study  of  23  buildings,  a  total  of 
1,497,275  ft2  of  conditioned  space  was  modeled.  This  yields  a  low  end  cost  of  $179,673  using 
the  value  of  $0.12  per  ft2  to  conduct  a  Level  2  audit  on  the  population  of  the  studied  REM 
buildings.  In  comparison,  applying  the  REM  process  to  this  population  of  buildings  yielded 
$0,005  per  ft2  for  a  total  cost  of  $6,900  to  conduct  the  REM  process  on  the  total  population  of  23 
buildings.  This  represents  a  cost  savings  of  96.17%. 

With  an  assumed  time  requirement  of  3  hours  per  building  (include  survey  collection,  modeling, 
and  energy  analysis),  REM  provides  an  opportunity  for  time  savings  (87.5%  low  end,  96.25% 
high  end)  compared  to  Level  2  auditing  approaches  that  may  require  3-10  days. 

From  this  demonstration  project,  the  REM  process  has  shown  to  be  useful  to  in  making  energy 
management  decisions.  The  REM  process  can  be  accomplished  with  personnel  with  less  exacting 
expertise  in  energy  systems,  saving  personnel  cost  and  increasing  the  ability  to  scale.  With  the 
number  of  individuals  doing  energy  assessments,  the  number  of  buildings  studied  can  also 
increase.  These  characteristics  of  REM  allow  this  process  to  be  more  cost  effective  then 
conducting  typical  Level  2  audits. 
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REM  may  also  precede  the  standard  energy  audit  process  to  act  as  a  triage  with  justifiable 
recommendations  to  select  the  high  priority  buildings  for  more  detailed  study.  With  the  ability  to 
compare  the  relative  merits  of  a  variety  of  ECMs,  the  REM  process  can  act  as  a  quick  proxy  for 
informing  installations  where  to  concentrate  building  life  cycle  cost  (BLCC)  studies  and  follow- 
on  detailed  actions. 
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8.0  IMPLEMENTATION  ISSUES 


To  facilitate  the  future  deployment  of  this  technology  the  following  topics  should  be  considered. 

8.1  METER  DATA  COMPARISONS 

There  were  several  issues  related  to  the  implementation  of  the  project,  specifically  comparison  of 
model  estimates  to  meter  data.  These  are  germane  in  the  execution  of  this  demonstration  pilot  as 
a  blind  study,  but  may  be  less  significant  in  future  deployments  of  this  technology  because  often 
the  meter  data  does  not  exist  or  is  unusable,  and  the  technical  confidence  for  this  approach  has 
been  demonstrated  by  this  project. 

The  results  of  this  project  recommend  REM  as  a  method  to  improve  DoD  building  data 
availability  considering  the  difficulty  with  the  current  building  energy  meter  deployments  at  the 
DoD.  REM  also  helped  identify  meters  that  were  not  functioning  correctly,  or  were  not  scaled 
correctly.  Meter  data  does  not  impact  the  recommendations  of  ECM  directly,  as  ECM 
recommendations  are  not  based  on  the  availability  of  meter  data.  If  meter  data  is  available,  it 
increases  the  transparency  of  the  energy  performance  of  a  building  and  can  supplement  the  ECM 
decision  process  in  support  of  the  REM  ECM  recommendations. 

There  are  numerous  concerns  with  the  quality  of  meter  data  at  DoD  buildings,  including 
common  issues  of  zero  readings,  time  gaps,  negative  readings,  large  jumps  or  jumps  in  usage, 
and  unknown  or  incorrect  scaling  of  meters.  There  were  also  other  issues  where  facility 
personnel  were  unaware  of  how  meters  were  divided  amongst  building(s),  how  to  access  interval 
data,  how  to  identify  or  correct  meter  scaling,  and  instances  where  staff  inaccurately  designated 
units,  or  the  meter  data  was  not  trusted  by  staff  familiar  with  the  building  (see  Appendix  D  in  the 
ESTCP  Final  Report  for  summary  of  meter  data  issues  with  individual  buildings). 

These  numerous  issues  draw  attention  to  the  need  for  DoD  to  review  data  from  existing  meters 
for  anomalies,  and  for  additional  and  perhaps  periodic  personnel  training  as  the  Advanced 
Metering  Initiative  (AMI)  continues  throughout  DoD  installations.  Future  research  using  the 
existing  data  set  should  compare  current  results  to  meter  data  after  smoothing  or  removing 
outlier  data. 

8.2  SITE  SELECTION  AND  DATA  GATHERING 

In  general,  installation  personnel  seem  stressed,  and  found  it  difficult  to  take  time  to  obtain 
metering  data  and  answer  building-related  questions. 

Several  installations  known  to  have  individually  metered  buildings  chose  not  to  participate  in 
REM  project  due  to  lack  of  manpower  or  previous  commitments.  The  government  sequester  that 
was  in  effect  during  the  study  period  may  have  added  to  these  resource  and  bandwidth  issues. 

Surveys  to  installation  POCs  contained  numerous  energy-related  questions  that  were  used  to  help 
researchers  interpret  results,  but  are  not  necessary  data  inputs  for  the  REM  workflows.  In  some 
cases,  suspected  misinformation  was  provided  in  the  survey  responses. 
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Data  requirements  for  the  model  can  be  streamlined  to  the  following  minimum  data 
requirements. 

•  Location  and  confirmation  that  building  is  visible  via  Google  satellite, 

•  Building  year  of  construction  and  major  renovation, 

•  Building  use  type, 

•  Operating  schedule, 

•  Gross  floor  area, 

•  Building  height  (whether  or  not  height  includes  unconditioned  attic  space  or  open  air, 
conditioned  spaces?), 

•  HVAC  system  type  or  selection  of  best  fit  from  Vasari  or  GBS  selection  options, 

•  Number  of  floors  (can  be  estimated  from  satellite,  if  unavailable), 

•  Floor  to  floor  height  (can  be  estimated  from  satellite,  if  unavailable), 

•  Percentage  window  glazing  (can  be  estimated  from  satellite/aerial  images,  if  unavailable), 

•  Percentage  skylight  glazing  (can  be  estimated  from  satellite,  if  unavailable), 

•  Exterior  wall  construction  and  insulation  levels  (can  be  estimated  from  satellite/aerial 
images  and  year  of  construction/renovation,  if  unavailable), 

•  Roof  construction  and  insulation  levels  (can  be  estimated  from  satellite  and  year  of 
construction/renovation,  if  unavailable), 

•  Glazing  type  and  skylight  types:  single  pane,  dual  pane,  triple  pane;  tinted,  low-e  (can  be 
estimated  based  on  year  of  construction/renovation  and  location  if  unknown),  and 

•  Documentation  of  known  structural  or  operational  idiosyncrasies. 

When  some  model  inputs  are  unknown,  assumptions  can  be  made  based  on  year  of  construction 
and/or  retrofit  and  satellite  images  from  Google  or  Bing.  If  a  user  wants  to  minimize  assumptions 
about  a  building,  they  either  need  knowledge  of  the  building  or  access  to  construction 
documents.  Construction  documents  would  also  provide  the  most  accurate  information  on 
building  floor  area.  The  use  of  as-built  documentation  was  initially  explored  as  a  method  for  data 
capture;  this  information  can  be  used  to  define  interior  zoning,  space  use  types,  mechanical/ 
electrical  system  design  and  building  envelope  design  but  given  the  time  required,  it  is  better 
suited  to  creation  of  detailed  energy  models,  not  REMS. 

While  many  assumptions  can  be  derived  from  satellite  images  and  on-site  visits  are  not  required 
for  the  REM  workflows,  it  is  helpful  to  have  knowledge  of  the  building,  access  to  building 
documentation  or  access  to  someone  with  knowledge  of  the  building  to  help  determine  model 
inputs. 
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8.3  ANALYTICAL  MODELING 


The  analysis  platform  and  workflow  do  not  allow  for  capturing  unregulated/process  loads.  Types 
of  spaces  with  high  energy  consumption  have  the  potential  to  throw  off  results.  Examples  may 
include:  labs,  data  centers,  or  kitchens.  And  process  load  types  such  as  exterior  lights,  elevators, 
lab  equipment,  or  machine  room  equipment  may  also  have  an  effect  on  end  use. 

It  should  also  be  noted  that  this  REM  methodology  using  the  DOE  2.2  engine  is  not  capable  of 
modeling  district  systems  such  as  steam  or  chilled  water,  nor  is  it  capable  of  modeling  multiple 
HVAC  systems,  radiant  heat,  or  heat  recovery  systems.  Sensitivity  in  the  models  is  derived  from 
limitations  in  modeling  or  understanding  of  the  spaces  such  as  attics,  basements,  atriums, 
unconditioned  spaces,  and  double  height  spaces.  There  may  also  be  impact  from  exterior 
obstructions  such  as  overhangs,  adjacent  buildings  and  solar  shading.  There  may  be  impact  from 
roof  zones  fabric  gains.  Additionally,  comprehending  the  operational  schedule  is  important  and 
useful  in  understanding  seasonal  variations  or  periods  of  non-use,  which  happens  frequently  for 
DoD  dormitories.  The  building  models  that  most  closely  followed  the  building  profiles  were  also 
the  closest  in  replicating  the  metered  energy  use,  and  it  is  possible  that  a  better  understanding  of 
operating  schedules,  seasonal  variations  in  usage  and  of  space  use  diversity,  could  have 
improved  results. 

The  recently  added  PES  feature  within  the  REM  software  (GBS)  allows  multiple  simultaneous 
energy  simulation  runs,  each  varying  values  for  building  features.  This  offers  significant  benefit 
in  that  it  automates  initial  exploration  and  identification  of  ECMs,  allowing  users  to  quickly  see 
which  building  parameters  have  the  most  influence  on  energy  consumption  and  the  highest 
opportunity  for  PES.  The  current  ESTCP  project  used  a  beta  version  of  the  PES  tool,  which  ran 
50  different  building  simulations.  The  production  version  since  released  utilizes  37  parameters 
and  tests  extreme  values  against  the  baseline  mode  in  the  initial  model.  This  format  can  provide 
teams  with  a  high  level  understanding  of  PES,  building  energy  performance  for  each  measured 
parameter,  and  can  provide  insight  on  building  sensitivity  to  various  parameters  of  the  buildings 
performance. 

GBS  has  the  capability  of  analyzing  renewable  energy  potential,  including  photovoltaic  and  wind 
energy,  and  can  also  calibrate  results  to  specific  weather  years  for  which  meter  data  is  available. 
A  government  satellite  blackout  in  the  fall  of  2012  prevented  researchers  from  calibrating  energy 
models  to  the  weather  year  in  GBS  and  manual  calibration  using  external  weather  data  files  was 
outside  of  the  scope  of  the  project.  Future  research  should  explore  calibration  to  actual  weather 
for  a  specific  year  and  document  buildings  that  would  be  best  for  renewable  energy 
implementation  based  on  assumed  installation  costs,  available  utility  rates,  modeled  geometry 
and  location. 

8.4  FUTURE  OF  REM 

Electric  results  were  consistently  higher  in  accuracy  than  gas  or  EUI  results  and  researchers 
recommend  further  exploration  around  gas  results.  The  project’s  performance  metrics  provided 
insight  on  accuracy  and  deviations;  however,  the  correlation  of  energy  use  curves  and  building 
use  categories  provide  greater  insight  on  accuracy  of  results  and  on  variations  throughout  the 
year.  Deviations  observed  between  meter  and  modeled  data  can  be  used  to  identify  buildings  that 
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are  not  operating  as  expected  and  should  be  prioritized  for  further  investigation  and  considered 
for  retrofits. 

REM  has  the  potential  to  help  DoD  scale  energy  assessments  across  the  building  portfolio, 
determine  buildings  in  the  portfolio  that  present  the  best  opportunity  for  retrofits,  quickly 
evaluate  relative  benefits  of  ECMs  through  auto-simulation  of  PES,  and  contribute  to  energy  and 
cost  savings  for  the  DoD. 

The  REM  workflows  allow  DoD  facility  managers  and  energy  managers  to  quickly  create 
building  models  based  on  limited  information,  rapidly  assess  buildings  that  are  using  the  most 
energy,  and  generate  reports.  Additionally,  using  the  PES  chart  and  automatic  simulations,  staff 
can  quickly  see  sensitivity  of  the  building  to  changes  in  parameters,  and  the  comparative  value  of 
modifications  to  HYAC,  roof,  walls,  windows,  lighting,  equipment,  etc.  REM  results  can  also 
help  DoD  make  informed  decisions  about  which  buildings  can  benefit  most  from  energy 
retrofits,  and  may  be  the  most  practical  to  meter  and  audit.  This  technology  can  help  DoD  to 
meet  existing  energy  auditing  and  energy  management  reporting  requirements  including  EISA 
2007. 

The  REM  workflow  is  easy  to  learn  and  DoD  facility  managers  can  generally  begin  creating 
energy  models  and  interpreting  results  after  3  hours  of  instruction.  REM  workflows  can  help 
scale  energy  analysis  throughout  the  DoD  at  a  pace  that  is  >90%  faster  and  95%  less  expensive 
than  ASHRAE  audits.  Initial  cultural  indications  are  that  this  method  is  well  received  at  the 
installations.  While  the  technology  is  new,  this  process  utilizes  a  category  software  tools  that  are 
familiar  to  installation  facility  asset  managers  (Google  Earth  and  computer  assisted  drawing 
[CAD]/BIM  software).  The  learning  curve  for  this  technology  is  measured  in  hours,  and  the 
startup  fees  are  low.  This  proves  that  this  technology  can  be  used  in  production  at  the 
installations  and  move  beyond  its  current  prototype  status. 

Deeper  investigations  may  include  applying  REM  across  more  climates  zones  and  building 
types,  comparisons  of  results  based  on  building  size  and  climate  zone,  or  examinations  of  results 
when  comparing  with  meter  data  at  intervals  versus  no  interval  data.  Studies  on  the  potential 
improved  accuracy  of  REM  when  using  smoothed  meter  data,  as  well  as  tracking  the  actual 
energy  savings  of  simulated  ECMs  to  the  actual  installed  energy  conservation  hardware  over 
time,  are  all  productive  areas  of  future  evaluations  for  this  REM  technology. 

Future  technical  studies  of  REM  may  prove  useful,  for  instance  examining  connections  to 
operational  asset  management  and  real  property  databases  systems  such  as  US  ACE  Builder 
software,  the  Military  Health  Service  Defense  Medical  Logistics  Standard  Support  System 
(DMLSS)  or  the  Air  Force  Geo-base  system.  With  these  systems,  operational,  material  and 
geometric  attributes  of  the  model  may  be  effectively  loaded  without  operator  input,  scripting  the 
data- loading  phase  could  scale  the  process  exceptionally.  With  integration  to  these  systems,  the 
REM  process  could  prove  more  efficient  by  working  within  the  context  of  the  daily  activities  of 
the  installation  and  would  allow  for  REM  analysis  on  the  entire  installation  at  once.  This  would 
allow  installations  to  have  EISA  type  reporting  information  for  the  entire  energy  modeled 
installation  inventory  each  year,  as  opposed  to  25%  annually  in  currently  mandates. 
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Brian  Parker 

Fort  Leonard  Wood 

1334  First  Street 

Building  2222 

Fort  Leonard  Wood,  MO  65473- 
8944 

Phone:  573-596-0901 

E-Mail:  bryan.l.parker.civ@mail.mil 

Installation  POC 

Peter 

Behrens 

Naval  Station  Great  Lakes 

2601  Paul  Jones  Street 

Great  Lakes,  IL  60088 

Phone:  847-688-2121  ext  28 

E-Mail:  peter.behrens@navy.mil 

Installation  POC 

Brian  Eckert 

Naval  Station  Great  Lakes 

2601  Paul  Jones  Street 

Great  Lakes,  IL  60088 

Phone:  847-204-2752 

E-Mail:  brian.eckert@navy.mil 

Installation  POC 

Sakhawat 

Amin 

JBLM 

Public  works  Joint  Base  Lewis 
McChord,  WA  98433-9500 

Phone:  253-966-9011 

E-Mail :  s akhawat .  amin.  ctr @mail .  mil 

Installation  POC 

Dave  Krohn 

JBLM 

Public  works  Joint  Base  Lewis 
McChord,  WA  98433-9500 

Phone:  253-966-1853 

E-Mail :  david.  a. krohn.  ci v@mail . mil 

Installation  POC 

Evelyn 

Baskin 

Panama  City;  NAVFAC  SE 

PWD  Panama  City 

101  Vernon  Avenue 

Building  126 

Panama  City,  FL  32407 

Phone:  850-230-7176 

E-Mail:  evelyn.baskin@navy.mil 

Installation  POC 

Randy 

Pieper 

Peterson  AFB 

21st  CES/CENP 

Peterson  AFB,  CO 

Phone:  719-556-9590 

E-Mail:  randall.pieper.ctr@us.af.mil 

Installation  POC 
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APPENDIX  A  (continued) 


POINTS  OF  CONTACT 


Point  of 
Contact 

Organization 

Phone 

E-Mail 

Role  In  Project 

Brian 

Ballweg 

Port  Hueneme 

Naval  Facilities  Engineering  and 
Expeditionary  Warfare  Center 
(NAVFAC  EXWC) 

Building  1100 

1100  23rd  Avenue 

Port  Hueneme,  CA  93043 

Phone:  805-982-1250 

E-Mail:  brian.ballweg@navy.mil 

Installation  POC 

Roberto 

Valdez 

Port  Hueneme 

Naval  Facilities  Engineering  and 
Expeditionary  Warfare  Center 
(NAVFAC  EXWC) 

Building  1100 

1100  23rd  Avenue 

Port  Hueneme,  CA  93043 

Phone:  805-982-1704 

E-Mail:  roberto.valdes@navy.mil 

Installation  POC 

Lance 

Mahar 

Portsmouth  Naval  Shipyard,  PWD 
Building  59/2 

Kittery,  ME  03904 

Phone:  207-438-5980 

E-Mail:  lance.mahar@navy.mil 

Installation  POC 

Elias 

Schtakleff 

Seymour  Johnson  AFB 

1510  Wright  Brothers  Avenue 
Goldsboro,  NC  27531 

Phone:  919-868-9179 

E-Mail: 

Elias.Schtakleff@seymourjohnson.af.mil 

Installation  POC 

Matthew 

Latham 

Seymour  Johnson  AFB 

1510  Wright  Brothers  Avenue 
Goldsboro,  NC  27531 

Phone:  919-722-7443 

E-Mail: 

Matthew.  Latham@seymourj  ohnson.  af.  mil 

Installation  POC 
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APPENDIX  B 


SUMMARY  DATA-  ALL  VIABLE  BUILDINGS:  OFFICES, 
BARRACKS,  SPECIALTY  USE,  COMBINED 


Building 

Number 

Building 

Type- 

Modeled 

Division 

Gross  Floor 

Area- 

Modeled 

Conditioned 

Space 

State 

kWh  From 

Meter 

kWh  from 
Model  (GBS) 

Mean  Bias 

Error- Model 

Vs  Meter  - 

Electric 

Accuracy 
Model  Vs 

Meter  - 

Absolute 

(Elec) 

MMBTU 

From  Meter 

MMBTU 

From 

Model 

(GBS) 

Mean  Bias 

Error-  Model 

Vs  Meter  - 

Gas 

Accuracy 
Model  Vs 

Meter 

Absolute 

(Gas) 

Total 

Metered 

EUI 

(kBtu/SF) 

Total 

Modeled 

EUI 

(kBtu/SF) 

Mean  Bias 

Error- 

Modeled  Vs 

Metered  - 

EUI 

(kBtu/SF) 

Accuracy 
Model  Vs 

Meter  - 

Absolute 

(EUI} 

B1 

Ofce 

Air  Force 

231,732 

CO 

5,599,413 

4,593,132 

-17,34% 

32.16% 

5524 

1279.54 

-77.09% 

23.09% 

37.33 

60.15 

-31.12% 

63.33% 

350 

Office 

Air  Force 

143,391 

CO 

1,723,137 

2,046,233 

13.41% 

31.59% 

2373.363 

1611369 

-43.99% 

56.19% 

53.95 

57.77 

-2.09% 

93.09% 

1100 

■Office 

Navy 

129,925 

CA 

1,195,292 

1,479,161 

33,33% 

66.17% 

179.1 

212.5311 

13.67% 

31.33% 

32.63 

43.51 

33.14% 

66.86% 

110 

Office 

Navy 

119,959 

FL 

1,763,209 

1,397,643 

7.32% 

92.63% 

2122,3 

443.371 

-79.99% 

29.91% 

63.52 

53,13 

-15,17% 

34.33% 

470 

Office 

Army 

191,595 

MO 

1,621,353 

1,564,214 

-3.55% 

96.45% 

2922.467 

1969.139 

-63.72% 

36.23% 

33.23 

0.09 

-24.35% 

75.65% 

■3309 

■Office 

Joint  Base 

59,573 

WA 

469,939 

569,359 

19.24% 

39.76% 

962.9 

593.6395 

-47.79% 

52.39% 

43.03 

40.55 

-5.37% 

94.13% 

Cell 

■Office 

Army 

52,739 

IL 

1,233,307 

1,062,475 

-17.56% 

32.44% 

4569.33 

1329.617 

-69.93% 

39.92% 

16933 

193.23 

-39.29% 

69.30% 

Cert2 

Office 

Army 

43,391 

IL 

979,952 

1,922,353 

4.33% 

95.62% 

4209.35 

2543.372 

-39.45% 

69.55% 

156.21 

124.93 

-29.92% 

79.93% 

531 

Office 

Navy 

49,237 

FL 

716,709 

694,433 

-15,66% 

34.34% 

736,1 

233.2419 

-69.69% 

39.31% 

39.23 

57.12 

-23.39% 

71.29% 

4421 

■Office 

Air  Force 

37,933 

NC 

706,325 

594,637 

-15,31% 

34.19% 

342 

259.2937 

-69.22% 

39.73% 

37.79 

61.71 

29  63% 

79.37% 

Cerl3 

Office 

Army 

23,639 

IL 

232,577 

279,563 

-1.97% 

99.93% 

2049.17 

696.9196 

-79.25% 

29.75% 

127.19 

66.04 

-43.94% 

51.96% 

1345 

Office  -  Bank 

Air  Force 

7,772 

CO 

113,197 

92,939 

-21.33% 

73.67% 

135.53 

131.9553 

-3.39% 

96,79% 

69.34 

57.70 

-16.79% 

33.21% 

1435 

Office  -  Bank 

Air  Force 

4,334 

CO 

57,639 

63,413 

9.94% 

99.96% 

142.1 

121.2332 

-14.63% 

35.32% 

79.12 

69.35 

-9.33% 

99.62% 

Summary  Data  for  Offices 

Average 

Accuracy 

35.70% 

Average 

Accuracy 

49.43% 

Average 

Accuracy 

77.34'% 

Mean 

Absolute 
Percentage 
Error  (MAPE) 

14.30% 

MAPE 

50.52% 

MAPE 

22  M% 

STDEV 

3.93% 

STDEV 

25.15% 

STDEV 

14.41% 

CoV 

10.41 

CoV 

50.34 

CoV 

13.63 

MFE 

2.01 

MFE 

13.31 

MFE 

21 

MAD 

7.21 

MAD 

13.35 

MAD 

22 

MSE 

70.23 

MSE 

684.33 

MSE 

913.06 
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APPENDIX  B  (continued) 

SUMMARY  DATA-  ALL  VIABLE  BUILDINGS:  OFFICES, 
BARRACKS,  SPECIALTY  USE,  COMBINED 


Building 

Number 

Building 

Type- 

Modeled 

Division 

Gross  Floor 

Area- 

Modeled 

Conditioned 

Space 

State 

kWh  From 

Meter 

kWh  from 
Model  (GBS) 

Mean  Bias 

Error- Model 

Vs  Meter  - 

Electric 

Accuracy 
Model  Vs 

Meter  - 

Absolute 

(Elec) 

MMBTU 

From  Meter 

MMBTU 

From 

Model 

(GBS) 

Mean  Bias 

Error-  Model 

Vs  Meter  - 

Gas 

Accuracy 
Model  Vs 

Meter 

Absolute 

(Gas) 

Total 

Metered 

EUI 

(kBtu/SF) 

Total 

Modeled 

EUI 

(kBtu/SF) 

Mean  Bias 

Error- 

Modeled  Vs 

Metered  - 

EUI 

(kBtu/SF) 

Accuracy 
Model  Vs 

Meter  - 

Absolute 

(EUI) 

831 

Barracks 

Army 

46,340 

MG 

943,960 

777,496 

-13.07% 

31.93% 

4257.623 

3593.201 

-15,61% 

34.39% 

133.56 

152.96 

-16,67% 

33.33% 

937 

Barracks 

Army 

55,724 

MO 

1,296,9 22 

380,220 

-32.13% 

67.37% 

4359.926 

3630.73 

-15,53% 

34.42% 

157.63 

119.97 

-23.92% 

76.08% 

434 

Barracks 

Navy 

96,130 

FL 

2,513,700 

2,693,241 

714% 

92.36% 

2203,8 

6162.932 

179.02% 

-79.02% 

112.22 

159.73 

42.33% 

57.67% 

9136 

Barracks 

Joint  Base 

25,349 

WA 

255,349 

310,415 

21.56% 

73.44% 

2732.3 

1669.32 

-33.92% 

61.03% 

142.19 

107.65 

-24.29% 

75.71% 

373 

Barracks 

Navy 

76,232 

ME 

353,400 

1,329,234 

54.36% 

45.14% 

4,621 

6200.039 

34.17% 

65.33% 

Ofl  OR 

140.75 

42.20% 

57.80% 

Summary  Data  for  Barracks 

Avg  Accuracy 

73.25% 

Average  w. 
outlier 

removed 

73.93% 

Avg  Accuracy 

70.12% 

MAPE 

26.75% 

MAPE 

26.07% 

MAPE 

29.33% 

STDEV 

13.07% 

STDEV 

12.25% 

STDEV 

11.71% 

CoV 

24.67 

CoV 

139.12 

CoV 

16.69 

MFE 

0.999 

MFE 

12.43 

MFE 

2.71 

MAD 

14.94 

MAD 

22.78 

MAD 

33.42 

MSE 

279.24 

MSE 

650.45 

MSE 

1510.53 

3650 

Cafeteria 

Air  Force 

23,013 

NC 

305,713 

636,023 

-21.06% 

73.94% 

2669.9 

2294.736 

-14.05% 

35.95% 

193.47 

159.41 

-17.61% 

32.39% 

4103 

Sdhod 

Air  Force 

25,351 

NC 

650,258 

470,667 

-27.62% 

72.33% 

0 

310.1262 

35.85 

74.14 

-13.64% 

36.36% 

4601 

Fi  restatic  n 

Air  Force 

43,137 

NC 

564,493 

746,343 

32  21% 

67.79% 

1,390 

731.9529 

-53.63% 

41.37% 

33.37 

77.09' 

-12.77% 

37.23% 

4537 

Automotive 

Fadty 

Air  Force 

33,700 

NC 

613,119 

549,983 

-10.30% 

39.70% 

2,203 

1251.493 

-43.19% 

56.31% 

111.00 

30.34 

-27.17% 

72.33% 

640 

Gym 

Army 

20,339 

MO 

237,632 

304,660 

5,90% 

94.10% 

2652.367 

2627.316 

-0.94% 

99.06% 

173.93 

175.55 

0.91% 

99.09% 

Summary  Data  for  Speciality  Use  Buildings 

Avg  Accuracy 

80.53% 

Avg  Accuracy 

70,30% 

Avg  Accuracy 

85.53% 

MAPE 

19.42% 

MAPE 

29.20% 

MAPE 

14.42% 

STDEV 

11. m 

STDEV 

26.39% 

STDEV 

9.47% 

CoV 

13,87 

CoV 

37.23 

CoV 

11.06 

MFE 

6.56 

MFE 

10.57 

MFE 

17.13 

MAD 

13.42 

MAD 

15.37 

MAD 

17.76 

EUI 

Summary  Data  for  All  Analyzed  Buildings 

Avg  Accuracy 

81.88% 

Avg  Accuracy 

53.20% 

Avg  Accuracy 

77.56% 

MAPE 

13.12% 

MAPE 

41,30% 

MAPE 

22.44% 

STDEV 

12.31% 

STDEV 

24.05% 

STDEV 

13.48% 

CoV 

15.03 

CoV 

43.33 

CoV 

17.33 

MFE 

2.73 

MFE 

13.30 

MFE 

16.03 

MAD 

10.24 

MAD 

19.75 

MAD 

2492 

MSE 

154.12 

MSE 

671.74 

MSE 

989.07 

APPENDIX  C 

MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Offices: 


Peterson  Building  1-  Gas  and  Electric 
Data 


•Metered  Gas  -kBtu 
•Modeled  Gas-  kBtu 
•Metered  Electric  -kBtu 
•Modeled  Electric-  kBtu 


Peterson  Building  350-  Gas  and  Electric 
Data 


Metered  Gas  -KBTU 
-Modeled  Gas-  kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


Fort  Leonard  Wood  Building  470  Gas 
Electric  Data 


Metered  Gas  -KBTU 
-Modeled  Gas-kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


Port  Hueneme  Building  1100  Gas  and 
Electric  Data 


-Metered  Gas  -kBTU 
-Modeled  Gas-  kBTU 
-Metered  Electric  -  KBTU 
-Modeled  Electric  -  KBTU 
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APPENDIX  C  (continued) 
MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Offices  (continued): 


Panama  City  Building  110  Gas  and 
Electric  Data 


•Metered  Gas  -KBTU 
-Modeled  Gas- kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


Seymour  Building  4421  Gas  and  Electric 
Data 


-Metered  Gas  -KBTU 
-Modeled  Gas- kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


JBLM  Building  3369  Gas  and  Electric  Data 


-Metered  Gas  -KBTU 
-Modeled  Gas-  kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


Peterson  Building  1345  Gas  and  Electric 
Data 


-Metered  Gas  -KBTU 
-Modeled  Gas- kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


APPENDIX  C  (continued) 
MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Offices  (continued): 


Panama  City  Building  581  Gas  and 
Electric  Data 


•Metered  Gas  -KBTU 
-Modeled  Gas-  kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


CERL  Building  1  Gas  and  Electric  Data 


-Metered  Gas  -KBTU 
-Modeled  Gas-  kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 


Peterson  Building  1485  Gas  and  Electric  Data 


-Metered  Gas  KBTU 
-Modeled  Gas- kBTU 
-Metered  Electric  -KBTU 
-Modeled  Electric-  fcRTL i 


v  /  ^  '  %^/// 
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APPENDIX  C  (continued) 
MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Offices  (continued): 


CERL  Building  3  Gas  and  Electric  Data 


-A- Metered  Gas  -KBTU 
-*- Modeled  Gas-  kBTU 
-♦-Metered  Electric  -KBTU 
—■—Modeled  Electric-  kBTU 


Barracks  Buildings: 

Dorm  484  Panama  City,  Florida 
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APPENDIX  C  (continued) 
MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Barracks  Buildings  (continued): 


Dorm  373  Portsmouth  Naval  Base 


Dorm  831  Fort  Leonard  Wood 
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APPENDIX  C  (continued) 
MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Barracks  Buildings  (continued): 
Dorm  937  Fort  Leonard  Wood 


Dorm  9136  Joint  Base  Lewis  McChord 


200,000 

180,000 

160,000 

140,000 

120,000 

100,000 

80,000 

00,000 

40,000 

20,000 

0 


ro 

3 

C 

ro 


ro 

3 
J 2 

II 


r. 


y 

ro 

2 


-Metered  Electric 
-Modeled  Electric 


KBTU 

kBTU 


-♦-Metered  Gas -KBTU 


-Modeled  Gas- kBTU 
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APPENDIX  C  (continued) 
MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Specialty  Use  Buildings: 


Cafeteria  3650,  Seymour  Air  Force  Base 


School  Building  4103,  Seymour  Air  Force  Base 


-Metered  Electric  -KBTU 
-Modeled  Electric-  kBTU 
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APPENDIX  C  (continued) 
MONTHLY  GAS  AND  ELECTRIC  CHARTS 


Specialty  Use  Buildings  (continued): 

Automotive  Facility  4537,  Seymour  Air  Force  Base 


Gym  Building  640,  Fort  Leonard  Wood 


